2023-02-16T16:18:14.239 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5 2023-02-16T16:18:14.240 INFO:teuthology.results:subset: '111/120000' 2023-02-16T16:18:14.240 INFO:teuthology.results:seed: '8577' 2023-02-16T17:47:46.365 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5 2023-02-16T17:47:46.394 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-02-17T05:39:43.806 INFO:teuthology.results:Tests finished! gathering results... 2023-02-17T05:39:53.546 INFO:scrape:Found 127 jobs 2023-02-17T05:39:56.296 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-2214-ga524676c (a524676c3a5885dae89d7cc602e846bac7b53313) reef (dev) , 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f47866aace0] , 2: gsignal() 2023-02-17T05:39:56.297 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T18:19:37.290 INFO:teuthology.task.internal: dump_program: ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce , 2023-02-16T18:19:37.291 DEBUG:teuthology.orchestra.run.smithi049:> which 'ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce' , /usr/bin/which: unrecognized option '--gtest_filter=*/2:-*SyntheticMatrixC*' 2023-02-17T05:39:56.308 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-2214-ga524676c (a524676c3a5885dae89d7cc602e846bac7b53313) reef (dev) , 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f47866aace0] , 2: gsignal() 2023-02-17T05:39:56.309 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T18:19:37.290 INFO:teuthology.task.internal: dump_program: ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce , 2023-02-16T18:19:37.291 DEBUG:teuthology.orchestra.run.smithi049:> which 'ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce' , /usr/bin/which: unrecognized option '--gtest_filter=*/2:-*SyntheticMatrixC*' 2023-02-17T05:39:56.319 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-2214-ga524676c (a524676c3a5885dae89d7cc602e846bac7b53313) reef (dev) , 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f47866aace0] , 2: gsignal() 2023-02-17T05:39:56.320 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T18:19:37.290 INFO:teuthology.task.internal: dump_program: ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce , 2023-02-16T18:19:37.291 DEBUG:teuthology.orchestra.run.smithi049:> which 'ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce' , /usr/bin/which: unrecognized option '--gtest_filter=*/2:-*SyntheticMatrixC*' 2023-02-17T05:39:56.330 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-2214-ga524676c (a524676c3a5885dae89d7cc602e846bac7b53313) reef (dev) , 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f47866aace0] , 2: gsignal() 2023-02-17T05:39:56.331 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T18:19:37.290 INFO:teuthology.task.internal: dump_program: ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce , 2023-02-16T18:19:37.291 DEBUG:teuthology.orchestra.run.smithi049:> which 'ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce' , /usr/bin/which: unrecognized option '--gtest_filter=*/2:-*SyntheticMatrixC*' 2023-02-17T05:39:56.341 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-2214-ga524676c (a524676c3a5885dae89d7cc602e846bac7b53313) reef (dev) , 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f47866aace0] , 2: gsignal() 2023-02-17T05:39:56.341 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T18:19:37.290 INFO:teuthology.task.internal: dump_program: ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce , 2023-02-16T18:19:37.291 DEBUG:teuthology.orchestra.run.smithi049:> which 'ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce' , /usr/bin/which: unrecognized option '--gtest_filter=*/2:-*SyntheticMatrixC*' 2023-02-17T05:39:56.351 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-2214-ga524676c (a524676c3a5885dae89d7cc602e846bac7b53313) reef (dev) , 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f47866aace0] , 2: gsignal() 2023-02-17T05:39:56.352 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T18:19:37.290 INFO:teuthology.task.internal: dump_program: ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce , 2023-02-16T18:19:37.291 DEBUG:teuthology.orchestra.run.smithi049:> which 'ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exce' , /usr/bin/which: unrecognized option '--gtest_filter=*/2:-*SyntheticMatrixC*' 2023-02-17T05:39:56.748 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:56.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:56.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:56.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:56.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:56.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:56.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:56.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:56.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:56.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:56.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:56.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:56.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:56.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:56.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:56.756 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:56.756 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:56.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:56.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:56.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:56.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:56.760 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:56.760 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:56.761 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:56.761 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:56.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:56.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:56.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:56.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:56.764 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.764 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:56.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:56.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:56.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:56.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:56.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:56.769 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.769 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:56.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:56.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:56.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:56.771 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:56.771 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:56.772 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:56.772 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:56.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:56.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:56.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:56.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:56.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:56.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:56.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:56.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:56.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.778 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.778 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:56.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:56.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:56.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:56.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:56.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:56.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:56.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:56.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:56.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:56.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:56.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:56.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:56.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:56.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:56.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:56.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:56.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:56.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:56.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:56.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:56.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:56.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:56.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:56.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:56.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:56.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:56.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:56.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:56.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:56.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:56.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:56.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:56.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:56.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:56.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:56.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:56.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:56.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.801 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.801 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:56.802 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:56.802 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:56.803 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:56.803 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:56.803 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:56.804 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:56.804 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:56.805 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:56.805 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:56.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:56.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:56.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:56.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:56.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:56.808 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:56.808 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:56.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:56.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:56.810 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:56.810 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:56.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:56.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:56.812 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:56.812 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:56.813 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:56.815 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:56.815 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:56.816 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:56.816 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:56.817 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:56.817 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:56.818 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:56.818 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:56.819 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:56.819 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:56.820 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:56.820 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:56.821 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:56.821 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:56.911 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:56.912 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:56.913 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:56.913 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:56.914 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:56.914 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:56.915 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:56.915 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:56.915 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:56.916 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:56.916 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:56.917 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:56.917 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:56.918 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:56.918 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:56.918 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:56.919 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:56.919 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:56.920 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.920 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:56.920 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.921 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.921 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:56.922 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:56.922 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:56.923 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:56.923 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:56.923 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:56.924 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.924 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:56.925 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:56.925 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:56.925 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:56.926 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.926 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.927 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.927 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.927 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:56.928 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.928 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:56.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:56.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:56.930 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:56.930 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:56.930 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.931 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:56.931 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:56.932 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:56.932 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:56.932 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:56.933 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:56.933 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:56.934 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:56.934 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:56.934 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:56.935 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:56.935 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:56.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:56.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:56.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:56.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.938 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.938 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:56.939 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.939 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.939 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.940 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:56.940 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:56.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:56.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:56.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:56.942 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:56.942 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:56.943 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:56.943 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:56.944 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:56.944 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:56.944 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:56.945 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.945 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:56.946 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:56.946 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:56.946 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.947 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:56.947 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:56.948 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:56.948 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:56.948 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:56.949 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:56.949 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.950 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.950 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:56.951 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:56.951 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:56.951 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:56.952 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:56.952 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:56.953 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:56.953 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.953 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.954 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.954 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.955 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:56.955 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:56.955 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:56.956 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:56.956 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.957 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:56.957 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:56.957 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:56.958 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:56.958 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:56.959 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:56.959 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:56.959 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:56.960 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:56.960 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:56.961 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:56.961 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:56.961 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:56.962 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:56.962 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:56.963 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:56.963 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:56.964 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:56.964 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:56.964 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:56.965 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:56.965 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:56.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:56.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:56.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:56.967 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:56.967 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:56.968 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:56.968 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:56.969 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:56.969 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:56.970 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:56.970 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:56.970 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:56.971 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:56.971 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:56.973 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:56.974 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:56.974 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:56.975 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:56.975 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:56.976 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:56.976 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:56.977 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:56.977 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:56.977 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:56.978 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:56.978 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:56.979 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:56.979 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:57.073 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:57.074 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:57.074 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:57.075 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:57.075 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.075 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:57.076 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:57.076 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:57.077 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:57.077 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.078 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:57.078 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:57.078 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:57.079 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:57.079 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:57.080 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:57.080 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:57.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:57.082 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.082 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.083 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.083 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:57.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:57.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:57.085 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:57.085 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.086 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:57.086 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:57.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:57.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:57.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.088 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.088 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.089 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.089 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:57.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:57.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:57.092 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:57.092 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.093 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:57.093 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.094 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:57.094 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:57.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:57.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:57.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:57.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:57.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:57.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:57.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:57.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:57.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:57.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.099 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:57.099 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.100 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.100 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:57.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.103 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:57.103 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:57.104 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.104 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.104 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.105 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.105 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.106 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:57.106 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:57.106 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:57.107 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:57.107 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.108 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:57.108 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:57.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:57.110 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:57.110 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:57.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.112 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.112 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.113 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:57.113 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:57.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:57.115 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.115 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.117 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.117 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.118 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:57.118 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:57.119 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:57.119 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.119 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.120 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:57.120 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.121 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:57.121 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:57.121 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:57.122 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:57.122 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:57.123 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.123 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.123 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.124 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:57.124 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:57.125 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:57.125 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.126 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:57.126 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:57.127 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:57.127 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:57.127 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:57.128 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:57.128 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:57.129 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.129 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:57.130 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:57.130 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:57.130 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:57.131 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:57.131 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.132 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.132 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.133 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:57.134 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:57.134 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:57.135 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.135 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.137 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.138 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:57.138 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:57.139 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:57.139 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:57.140 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:57.140 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.141 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:57.141 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.142 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.142 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:57.143 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:57.143 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:57.144 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:57.242 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:57.243 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:57.243 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:57.243 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:57.245 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.245 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:57.246 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:57.246 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:57.247 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:57.247 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.247 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:57.248 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:57.248 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:57.249 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:57.249 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:57.250 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:57.250 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.250 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:57.251 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.251 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:57.252 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.252 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.252 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.253 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:57.253 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:57.254 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.254 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:57.255 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:57.255 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.255 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:57.256 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:57.256 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:57.257 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:57.257 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.257 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.258 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.258 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.259 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:57.259 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.260 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.260 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:57.260 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.261 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:57.261 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:57.262 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.262 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:57.262 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.263 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:57.263 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:57.264 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:57.264 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:57.264 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:57.265 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:57.265 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:57.266 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:57.266 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:57.266 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:57.267 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:57.267 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:57.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.269 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.269 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:57.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:57.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:57.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.273 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.273 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.274 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.274 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:57.275 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:57.275 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:57.275 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:57.276 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.276 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:57.277 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.277 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:57.277 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.278 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:57.278 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:57.279 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.279 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:57.279 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.280 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.280 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.281 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.281 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:57.281 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.282 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:57.282 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.283 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:57.283 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.283 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.284 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.284 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.285 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.285 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.285 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.286 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:57.286 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:57.287 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:57.287 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.287 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.288 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:57.288 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.289 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:57.289 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:57.289 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:57.290 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:57.290 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:57.291 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.291 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.291 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.292 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:57.292 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:57.293 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:57.293 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.294 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:57.294 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:57.294 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:57.295 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:57.295 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:57.296 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:57.296 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:57.296 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.297 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:57.297 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:57.298 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:57.298 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:57.298 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:57.299 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.299 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.300 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.300 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:57.301 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:57.301 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:57.302 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.302 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.304 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.304 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:57.305 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:57.305 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:57.306 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:57.306 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:57.307 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.307 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:57.308 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.308 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.309 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:57.309 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:57.310 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:57.310 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:57.397 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:57.399 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:57.399 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:57.399 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:57.400 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.400 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:57.401 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:57.402 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:57.402 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:57.402 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.403 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:57.403 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:57.404 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:57.404 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:57.405 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:57.405 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:57.405 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.406 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:57.406 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.407 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:57.407 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.408 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.408 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.408 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:57.409 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:57.409 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.410 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:57.410 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:57.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:57.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:57.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:57.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:57.413 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.413 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:57.415 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.415 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.416 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:57.416 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:57.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:57.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.418 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:57.418 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.419 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:57.419 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:57.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:57.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:57.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:57.421 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:57.421 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:57.422 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:57.422 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:57.423 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:57.423 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:57.424 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.424 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:57.424 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.425 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.425 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.426 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:57.426 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.427 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.427 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.427 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.428 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:57.428 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:57.429 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.429 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.430 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.430 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.431 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.431 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:57.431 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:57.432 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:57.432 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:57.433 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.433 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:57.434 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.434 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:57.434 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.435 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:57.435 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:57.436 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.436 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:57.436 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.437 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.437 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.438 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.438 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:57.439 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.439 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:57.439 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.440 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:57.440 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.441 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.441 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.442 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.442 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.442 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.443 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.443 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:57.444 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:57.444 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:57.445 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.445 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.445 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:57.446 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.446 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:57.447 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:57.447 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:57.448 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:57.448 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:57.448 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.449 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.449 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.450 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:57.450 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:57.451 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:57.451 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.451 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:57.452 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:57.452 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:57.453 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:57.453 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:57.454 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:57.454 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:57.454 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.455 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:57.455 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:57.456 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:57.456 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:57.457 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:57.457 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.458 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.458 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.459 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:57.459 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:57.460 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:57.460 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.460 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.462 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.463 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:57.463 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:57.464 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:57.464 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:57.465 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:57.465 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.466 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:57.466 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.467 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.467 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:57.468 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:57.468 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:57.469 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:57.555 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:57.556 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:57.557 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:57.557 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:57.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:57.559 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:57.559 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:57.559 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:57.560 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.560 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:57.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:57.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:57.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:57.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:57.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:57.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:57.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:57.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.565 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.565 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.566 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:57.566 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:57.566 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.567 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:57.567 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:57.568 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.568 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:57.568 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:57.569 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:57.570 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:57.570 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.570 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.571 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.571 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:57.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.573 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:57.573 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:57.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:57.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.575 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:57.575 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.576 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:57.576 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:57.577 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:57.577 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:57.577 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:57.578 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:57.578 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:57.579 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:57.579 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:57.580 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:57.580 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:57.580 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.581 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:57.581 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.582 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.582 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.582 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:57.583 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.583 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:57.585 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:57.585 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.586 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.586 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.587 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.587 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.587 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:57.588 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:57.588 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:57.589 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:57.589 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.590 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:57.590 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.590 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:57.591 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.591 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:57.592 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:57.592 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.592 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:57.593 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.593 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.594 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.594 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.594 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:57.595 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.595 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:57.596 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.596 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:57.596 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.597 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.597 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.598 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.598 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.598 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.599 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.599 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:57.600 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:57.600 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:57.601 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.601 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.602 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:57.602 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.602 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:57.603 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:57.603 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:57.604 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:57.604 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:57.604 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.605 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.605 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.606 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:57.606 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:57.607 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:57.607 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.607 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:57.608 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:57.608 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:57.609 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:57.609 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:57.610 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:57.610 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:57.610 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.611 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:57.611 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:57.612 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:57.612 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:57.612 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:57.613 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.613 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.614 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.614 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:57.615 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:57.615 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:57.616 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.616 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.618 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.619 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:57.619 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:57.620 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:57.620 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:57.620 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:57.621 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.621 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:57.622 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.622 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.623 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:57.623 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:57.624 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:57.624 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:57.724 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-857-ga2cdaa1a (a2cdaa1abe0a19cb94cfaf47f6c002401fd39acb) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f20aa9f5ae3] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12ca) [0x556f59bdea8a] 2023-02-17T05:39:57.725 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:31 smithi099 ceph-mon[105223]: osdmap e666: 12 total, 12 up, 9 in , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: pgmap v1122: 77 pgs: 77 active+clean; 674 MiB data, 1.5 GiB used, 803 GiB / 805 GiB avail; 3.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 2.0 MiB/s, 1 objects/s recovering , 2023-02-16T17:32:32.334 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:31 smithi150 ceph-mon[107318]: from='mgr.14153 172.21.15.99:0/1879354849' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "56"}]': finished 2023-02-17T05:39:57.726 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", , 2023-02-16T17:32:32.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819817+0000", 2023-02-17T05:39:57.726 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", , 2023-02-16T17:32:32.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:25:35.819957+0000", 2023-02-17T05:39:57.727 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:32.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.727 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2023-02-17T05:39:57.727 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:32.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:32.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2023-02-17T05:39:57.728 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4294967295.9999986 , 2023-02-16T17:32:33.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-02-17T05:39:57.728 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8089999999999998e-06 , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: } , 2023-02-16T17:32:33.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-02-17T05:39:57.729 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.927232+0000", , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00062145799999999997 , 2023-02-16T17:32:33.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-02-17T05:39:57.729 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.050 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24532.0:4876 2.19 2.fe0488f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e475)", 2023-02-17T05:39:57.730 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:20.941100+0000", , 2023-02-16T17:32:33.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4134999999999999e-05 2023-02-17T05:39:57.730 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.03876794999999, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98452678299999996, , 2023-02-16T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2023-02-17T05:39:57.731 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", , 2023-02-16T17:32:33.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2023-02-16T17:29:41.736060+0000", 2023-02-17T05:39:57.731 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: { , 2023-02-16T17:32:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2023-02-17T05:39:57.731 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:{ , 2023-02-16T17:32:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], , 2023-02-16T17:32:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2023-02-17T05:39:57.732 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.733 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,8", 2023-02-17T05:39:57.733 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.733 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24532.0:5029 2.d 2:b250af82:::smithi150130474-26:head [copy-from ver 740] snapc 17e=[17e,17c,179,176,175,174,173,16d,167,166,163,162,158,153,146,143] ondisk+write+known_if_redirected+supports_pool_eio e486)", , 2023-02-16T17:32:39.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:29:31.868849+0000", 2023-02-17T05:39:57.734 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:31.868895+0000", , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:39.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.734 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:53.956706+0000", , 2023-02-16T17:32:39.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9313999999999998e-05 , 2023-02-16T17:32:39.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:54.890457+0000", , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366999999999995e-05 , 2023-02-16T17:32:39.610 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:22.595556+0000", , 2023-02-16T17:32:39.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9410000000000002e-06 2023-02-17T05:39:57.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:26.879619+0000", , 2023-02-16T17:32:39.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024250899999999999 2023-02-17T05:39:57.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.519477439, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91042740700000002, , 2023-02-16T17:32:39.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", , 2023-02-16T17:32:39.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:59.983808+0000", 2023-02-17T05:39:57.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:39.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:31:00.374585+0000", 2023-02-17T05:39:57.738 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:39.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.738 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:39.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2023-02-17T05:39:57.738 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767444259}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4250538715}) , 2023-02-16T17:32:42.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:41 smithi099 ceph-mon[105223]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273337691}) 2023-02-17T05:39:57.739 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1791531713}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=44585003}) , 2023-02-16T17:32:42.343 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:41 smithi150 ceph-mon[107318]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2714261775}) 2023-02-17T05:39:57.739 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1917080815' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch , 2023-02-16T17:32:43.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:42 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3456658170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2023-02-17T05:39:57.740 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000151244 , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.740 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3424999999999994e-05 , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.741 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44827122400000002 , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.741 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7314e-05 , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.742 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4158 , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2023-02-17T05:39:57.742 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:19.841053+0000", , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8600000000000003e-06 , 2023-02-16T17:32:45.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.743 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064146+0000", , 2023-02-16T17:32:45.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.743 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:09.064170+0000", , 2023-02-16T17:32:45.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6019999999999998e-06 2023-02-17T05:39:57.744 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.124322009, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63729258200000005, , 2023-02-16T17:32:45.712 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2023-02-17T05:39:57.744 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", , 2023-02-16T17:32:45.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:30:24.564516+0000", 2023-02-17T05:39:57.744 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:45.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2023-02-17T05:39:57.745 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:45.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.745 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.571e-06 , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:45.764 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-02-17T05:39:57.746 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:35.973873+0000", , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023747699999999999 , 2023-02-16T17:32:45.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.746 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: purged_snaps scrub ok , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: 1.2 scrub starts , 2023-02-16T17:32:51.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:32:50 smithi099 ceph-mon[105223]: osdmap e685: 12 total, 12 up, 12 in 2023-02-17T05:39:57.747 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:51.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.708319+0000", , 2023-02-16T17:32:51.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92655564599999996 2023-02-17T05:39:57.747 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", , 2023-02-16T17:32:51.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:35.802043+0000", 2023-02-17T05:39:57.747 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", , 2023-02-16T17:32:51.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:25:36.761867+0000", 2023-02-17T05:39:57.748 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2023-02-16T17:25:51.878049+0000", , 2023-02-16T17:32:51.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 420.024327164, , 2023-02-16T17:32:51.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022400824999999999, 2023-02-17T05:39:57.748 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2023-02-17T05:39:57.749 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:51.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2023-02-17T05:39:57.749 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.013 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2023-02-17T05:39:57.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2023-02-17T05:39:57.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2023-02-17T05:39:57.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2023-02-17T05:39:57.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.081 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:52.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2023-02-17T05:39:57.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.086213636999999996 , 2023-02-16T17:32:52.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000231303 , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:52.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:44.921001+0000", , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 , 2023-02-16T17:32:52.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub starts , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.2a scrub ok , 2023-02-16T17:32:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:54 smithi122 ceph-mon[106988]: 2.1d scrub starts 2023-02-17T05:39:57.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061259800000000001 , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:13.747308+0000", , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.229445814 , 2023-02-16T17:32:58.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:28:21.482432+0000", , 2023-02-16T17:32:58.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2023-02-17T05:39:57.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:18.696471+0000", , 2023-02-16T17:32:58.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055018799999999996 2023-02-17T05:39:57.756 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", , 2023-02-16T17:32:58.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:29:41.591302+0000", 2023-02-17T05:39:57.756 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:58.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2023-02-17T05:39:57.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: } , 2023-02-16T17:32:58.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] , 2023-02-16T17:32:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-02-17T05:39:57.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3244999999999993e-05 , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, , 2023-02-16T17:32:58.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2023-02-17T05:39:57.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:50.905588+0000", , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050961668000000002 , 2023-02-16T17:32:59.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-02-17T05:39:57.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.033989+0000", , 2023-02-16T17:32:59.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: { , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", , 2023-02-16T17:32:59.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2023-02-16T17:32:51.203465+0000", 2023-02-17T05:39:57.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:00.335 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:32:59 smithi150 ceph-mon[107318]: 2.2b scrub ok , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub starts , 2023-02-16T17:33:00.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:32:59 smithi122 ceph-mon[106988]: 2.c scrub ok 2023-02-17T05:39:57.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub starts , 2023-02-16T17:33:03.180 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:02 smithi150 ceph-mon[107318]: 2.11 scrub ok , 2023-02-16T17:33:03.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:02 smithi122 ceph-mon[106988]: purged_snaps scrub starts 2023-02-17T05:39:57.760 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.386 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:04 smithi122 ceph-mon[106988]: 1.3 scrub ok , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:{ , 2023-02-16T17:33:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, 2023-02-17T05:39:57.760 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.072768+0000", , 2023-02-16T17:33:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9932999999999997e-05 , 2023-02-16T17:33:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.761 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_op(client.24532.0:5518 2.f 2:f24a46f3:::smithi150130474-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e536)", 2023-02-17T05:39:57.761 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.075801+0000", , 2023-02-16T17:33:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4294967295.9999995 2023-02-17T05:39:57.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:30:22.294378+0000", , 2023-02-16T17:33:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0044664029999999999 2023-02-17T05:39:57.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:00.372290+0000", 2023-02-17T05:39:57.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "header_read", 2023-02-17T05:39:57.764 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.764 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "op_commit", 2023-02-17T05:39:57.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.2239999999999997e-06 , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.0038627399999999999 , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type_data": { , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flag_point": "reached pg", , 2023-02-16T17:33:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ 2023-02-17T05:39:57.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:08.082 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub starts , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: 1.15 scrub ok , 2023-02-16T17:33:08.083 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:07 smithi150 ceph-mon[107318]: pgmap v1169: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 669 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "size": 20, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 600, , 2023-02-16T17:33:12.071 INFO:teuthology.orchestra.run.smithi150.stdout: "ops": [ 2023-02-17T05:39:57.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.085 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", 2023-02-17T05:39:57.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", 2023-02-17T05:39:57.769 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.112 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", 2023-02-17T05:39:57.769 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.125 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00025275900000000002 , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 5.9038e-05 , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.152 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000109302 , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.165 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.771 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:25:59.916190+0000", , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:12.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.771 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:01.780419+0000", , 2023-02-16T17:33:12.191 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.772 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.982861+0000", , 2023-02-16T17:33:12.204 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 9.9525999999999995e-05 2023-02-17T05:39:57.772 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "sub_op_commit_rec", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:02.991875+0000", , 2023-02-16T17:33:12.217 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00054520999999999997 2023-02-17T05:39:57.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.705311+0000", 2023-02-17T05:39:57.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: ] , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:12.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:12.256 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "started", 2023-02-17T05:39:57.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:16.064 INFO:teuthology.orchestra.run.smithi150.stdout:} , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub starts , 2023-02-16T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:16 smithi150 ceph-mon[107318]: 1.1b scrub ok 2023-02-17T05:39:57.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.075 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:23:51.846109+0000", , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00014438100000000001 , 2023-02-16T17:33:18.076 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.089 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:22.798488+0000", , 2023-02-16T17:33:18.090 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 7.165e-06 2023-02-17T05:39:57.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:18.102 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:26:23.797909+0000", , 2023-02-16T17:33:18.103 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.99916530000000003 2023-02-17T05:39:57.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "commit_sent", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:27:08.074918+0000", , 2023-02-16T17:33:18.115 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 4.1551000000000002e-05 2023-02-17T05:39:57.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:18.128 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:41.707454+0000", 2023-02-17T05:39:57.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.141 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", 2023-02-17T05:39:57.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:18.154 INFO:teuthology.orchestra.run.smithi150.stdout: { 2023-02-17T05:39:57.778 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.113354+0000", , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 , 2023-02-16T17:33:18.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-02-17T05:39:57.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.285637+0000", , 2023-02-16T17:33:18.180 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0 2023-02-17T05:39:57.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", , 2023-02-16T17:33:18.194 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:46.457175+0000", 2023-02-17T05:39:57.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "events": [ , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:18.207 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "initiated", 2023-02-17T05:39:57.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: pgmap v1180: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub starts , 2023-02-16T17:33:18.294 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:18 smithi150 ceph-mon[107318]: 1.5 scrub ok 2023-02-17T05:39:57.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:22.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: pgmap v1183: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 672 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: osdmap e701: 12 total, 12 up, 12 in , 2023-02-16T17:33:22.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:22 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/1437259528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.142 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "osd_repop(osd.4.0:852 2.26 e611/557 2:66d7780c:::smithi150130474-28:1e7 v 611'893, mlcod=610'887)", 2023-02-17T05:39:57.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "queued_for_pg", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:58.911714+0000", , 2023-02-16T17:33:24.157 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 3.766e-06 2023-02-17T05:39:57.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "reached_pg", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:31:59.192096+0000", , 2023-02-16T17:33:24.170 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.27899320799999999 2023-02-17T05:39:57.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.183 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "all_read", , 2023-02-16T17:33:24.184 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:35.918851+0000", 2023-02-17T05:39:57.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.197 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "throttled", 2023-02-17T05:39:57.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.209 INFO:teuthology.orchestra.run.smithi150.stdout: "description": "MOSDPGRecoveryDelete(2.31 e670,668 2:8ef88a11:::smithi150130474-30:1b2,569'437)", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "initiated_at": "2023-02-16T17:32:36.274132+0000", , 2023-02-16T17:33:24.210 INFO:teuthology.orchestra.run.smithi150.stdout: "age": 47.849135187000002, 2023-02-17T05:39:57.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00023079 , 2023-02-16T17:33:24.222 INFO:teuthology.orchestra.run.smithi150.stdout: } , 2023-02-16T17:33:24.223 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-02-17T05:39:57.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.730343+0000", , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.000152888 , 2023-02-16T17:33:24.235 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-02-17T05:39:57.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:49.909156+0000", , 2023-02-16T17:33:24.249 INFO:teuthology.orchestra.run.smithi150.stdout: "duration": 0.00029398299999999999 2023-02-17T05:39:57.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", , 2023-02-16T17:33:24.262 INFO:teuthology.orchestra.run.smithi150.stdout: "time": "2023-02-16T17:32:50.073920+0000", 2023-02-17T05:39:57.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: { , 2023-02-16T17:33:24.275 INFO:teuthology.orchestra.run.smithi150.stdout: "event": "done", 2023-02-17T05:39:57.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2243078855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished , 2023-02-16T17:33:26.384 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: osdmap e704: 12 total, 12 up, 12 in , 2023-02-16T17:33:26.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:26 smithi122 ceph-mon[106988]: 2.e scrub starts 2023-02-17T05:39:57.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2023-02-16T17:33:37.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:37 smithi099 ceph-mon[105223]: osdmap e709: 12 total, 12 up, 12 in , 2023-02-16T17:33:37.584 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:33:37 smithi150 ceph-mon[107318]: from='client.? 172.21.15.99:0/1278883550' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2023-02-17T05:39:57.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:45.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:45 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/3168063283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config , 2023-02-16T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:33:46 smithi099 ceph-mon[105223]: pgmap v1208: 87 pgs: 87 active+clean; 692 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub starts , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: 1.6 scrub ok , 2023-02-16T17:33:57.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:33:57 smithi122 ceph-mon[106988]: from='client.? 172.21.15.99:0/2821640389' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: 2.0 scrub ok , 2023-02-16T17:34:06.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: pgmap v1230: 87 pgs: 87 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s, 6 objects/s recovering , 2023-02-16T17:34:06.305 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:05 smithi099 ceph-mon[105223]: from='client.? 172.21.15.99:0/1092054470' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2023-02-17T05:39:57.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.6 flush_pg_stats , 2023-02-16T17:34:10.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.7 flush_pg_stats , 2023-02-16T17:34:10.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a524676c3a5885dae89d7cc602e846bac7b53313 shell --fsid 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-02-17T05:39:57.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: pgmap v1237: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.2a scrub ok , 2023-02-16T17:34:16.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:16 smithi150 ceph-mon[107318]: 2.4 scrub starts 2023-02-17T05:39:57.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 2.2 scrub ok , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub starts , 2023-02-16T17:34:25.304 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 16 17:34:25 smithi099 ceph-mon[105223]: 1.2 deep-scrub ok 2023-02-17T05:39:57.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.37 scrub ok , 2023-02-16T17:34:29.586 INFO:journalctl@ceph.mon.c.smithi150.stdout:Feb 16 17:34:29 smithi150 ceph-mon[107318]: 2.23 scrub starts , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:33.925 INFO:teuthology.orchestra.run.smithi099.stdout:300647710884 , 2023-02-16T17:34:34.261 INFO:teuthology.orchestra.run.smithi099.stdout:107374182589 , Inferring config /var/lib/ceph/6cc4bd56-ae1d-11ed-9ae8-001a4aab830c/mon.a/config 2023-02-17T05:39:57.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.8 scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: 2.2b scrub starts , 2023-02-16T17:34:38.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:37 smithi122 ceph-mon[106988]: pgmap v1251: 87 pgs: 1 active+clean+inconsistent, 86 active+clean; 676 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-02-17T05:39:57.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub starts , 2023-02-16T17:34:45.385 INFO:journalctl@ceph.mon.b.smithi122.stdout:Feb 16 17:34:44 smithi122 ceph-mon[106988]: 2.f deep-scrub ok , 2023-02-16T17:34:45.998 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-02-17T05:39:57.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:34:49.184 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:48 smithi099 systemd[1]: Stopping Ceph mgr.y for 6cc4bd56-ae1d-11ed-9ae8-001a4aab830c... , 2023-02-16T17:34:49.484 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 16 17:34:49 smithi099 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-mgr-y[105469]: 2023-02-16T17:34:49.173+0000 7fdfd9e58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 , 2023-02-16T17:34:49.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c@mgr.y.service' 2023-02-17T05:39:57.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:24.635 INFO:journalctl@ceph.osd.5.smithi122.stdout:Feb 16 17:35:24 smithi122 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-5[116964]: 2023-02-16T17:35:24.182+0000 7f53dc488700 -1 osd.5 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:52.092643+0000 front 2023-02-16T17:34:52.092487+0000 (oldest deadline 2023-02-16T17:35:17.392671+0000) , 2023-02-16T17:35:25.085 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:24.791+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:55.082198+0000 front 2023-02-16T17:34:55.082040+0000 (oldest deadline 2023-02-16T17:35:20.982129+0000) , 2023-02-16T17:35:25.086 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:24 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:24.893+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) 2023-02-17T05:39:57.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:32.267 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:31 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:31.933+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:32.585 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:35:32 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:35:32.254+0000 7fdcb5a28700 -1 osd.10 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.860113+0000 front 2023-02-16T17:34:54.661215+0000 (oldest deadline 2023-02-16T17:35:16.760258+0000) 2023-02-17T05:39:57.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.8.smithi150.stdout:Feb 16 17:35:37 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-8[167154]: 2023-02-16T17:35:37.926+0000 7fbd03d21700 -1 osd.8 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:07.984083+0000 front 2023-02-16T17:35:07.984247+0000 (oldest deadline 2023-02-16T17:35:30.884213+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.142703+0000 front 2023-02-16T17:34:53.142873+0000 (oldest deadline 2023-02-16T17:35:19.042691+0000) , 2023-02-16T17:35:38.273 INFO:journalctl@ceph.osd.9.smithi150.stdout:Feb 16 17:35:38 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-9[117772]: 2023-02-16T17:35:38.014+0000 7f82610be700 -1 osd.9 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:01.944033+0000 front 2023-02-16T17:35:01.944125+0000 (oldest deadline 2023-02-16T17:35:26.043995+0000) 2023-02-17T05:39:57.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:43.087 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:42 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:42.805+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:48.251 DEBUG:teuthology.orchestra.run:got remote process result: None , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.7:Stopped osd.7 , 2023-02-16T17:35:48.252 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-02-17T05:39:57.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:53.062 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:52 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:52.712+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2023-02-16T17:35:00.950566+0000 front 2023-02-16T17:35:00.950807+0000 (oldest deadline 2023-02-16T17:35:25.050641+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2023-02-16T17:35:05.051213+0000 front 2023-02-16T17:35:05.051061+0000 (oldest deadline 2023-02-16T17:35:30.951166+0000) , 2023-02-16T17:35:58.086 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:35:57 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:35:57.822+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) 2023-02-17T05:39:57.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:03.588 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Got signal Terminated *** , 2023-02-16T17:36:03.589 INFO:journalctl@ceph.osd.10.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-10[169276]: 2023-02-16T17:36:03.225+0000 7fdcbcc39700 -1 osd.10 730 *** Immediate shutdown (osd_fast_shutdown=true) *** , 2023-02-16T17:36:04.085 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:03 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:03.812+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2023-02-16T17:34:53.949708+0000 front 2023-02-16T17:34:53.949814+0000 (oldest deadline 2023-02-16T17:35:16.249641+0000) 2023-02-17T05:39:57.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:11.336 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2023-02-16T17:35:15.552741+0000 front 2023-02-16T17:35:15.552802+0000 (oldest deadline 2023-02-16T17:35:38.452695+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2023-02-16T17:35:24.254253+0000 front 2023-02-16T17:35:24.254233+0000 (oldest deadline 2023-02-16T17:35:46.554125+0000) , 2023-02-16T17:36:11.337 INFO:journalctl@ceph.osd.11.smithi150.stdout:Feb 16 17:36:10 smithi150 ceph-6cc4bd56-ae1d-11ed-9ae8-001a4aab830c-osd-11[127092]: 2023-02-16T17:36:10.836+0000 7fcfa1ef9700 -1 osd.11 730 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2023-02-16T17:35:26.554714+0000 front 2023-02-16T17:35:26.554538+0000 (oldest deadline 2023-02-16T17:35:52.454629+0000) 2023-02-17T05:39:57.799 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2023-02-17T05:39:57.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-02-16T17:36:38.767 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi099 , 2023-02-16T17:36:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2023-02-16T17:36:38.825 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-02-16_16:16:31-rados-wip-yuri7-testing-2023-02-08-1414-distro-default-smithi/7176378/remote/smithi122 2023-02-17T05:39:58.333 INFO:scrape:Found 8 distinct failure reasons 2023-02-17T05:39:58.334 INFO:scrape:Failure: Command failed on smithi149 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-02-17T05:39:58.335 INFO:scrape:6 jobs: ['7176388', '7176424', '7176334', '7176325', '7176367', '7176362'] 2023-02-17T05:39:58.335 INFO:scrape:suites intersection: [] 2023-02-17T05:39:58.336 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', 'agent/on', 'ceph', 'cluster/1-node', 'cluster/3-node', 'clusters/{fixed-2', 'fast/normal', 'k8s/1.21', 'mon_election/connectivity', 'msgr-failures/fastclose', 'net/calico', 'net/flannel', 'net/host', 'openstack}', 'rados', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-recovery}', 'rook/1.7.2}', 'rook/master}', 'supported-random-distro$/{ubuntu_latest}', 'task/test_iscsi_pids_limit/{centos_8.stream_container_tools', 'test_iscsi_pids_limit}}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-small-objects-fast-read-overwrites}'] 2023-02-17T05:39:58.336 INFO:scrape: 2023-02-17T05:39:58.336 INFO:scrape:Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi181 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a524676c3a5885dae89d7cc602e846bac7b53313 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' 2023-02-17T05:39:58.337 INFO:scrape:2 jobs: ['7176340', '7176330'] 2023-02-17T05:39:58.337 INFO:scrape:suites intersection: ['mon_election/classic', 'msgr-failures/none', 'msgr/async-v2only', 'rados', 'supported-random-distro$/{ubuntu_latest}}', 'test_envlibrados_for_rocksdb}'] 2023-02-17T05:39:58.337 INFO:scrape:suites union: ['mon_election/classic', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/centos_latest', 'rados/singleton/{all/test_envlibrados_for_rocksdb/{supported/rhel_latest', 'supported-random-distro$/{ubuntu_latest}}', 'test_envlibrados_for_rocksdb}'] 2023-02-17T05:39:58.338 INFO:scrape: 2023-02-17T05:39:58.338 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a524676c3a5885dae89d7cc602e846bac7b53313 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-02-17T05:39:58.338 INFO:scrape:2 jobs: ['7176423', '7176333'] 2023-02-17T05:39:58.339 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] 2023-02-17T05:39:58.339 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-hybrid}', 'random-objectstore$/{filestore-xfs}', 'tasks/e2e}'] 2023-02-17T05:39:58.339 INFO:scrape: 2023-02-17T05:39:58.339 INFO:scrape:Dead: b'2023-02-17T04:26:09.956 DEBUG:teuthology.exit:Finished running handlers' 2023-02-17T05:39:58.340 INFO:scrape:6 jobs: ['7176326', '7176329', '7176328', '7176341', '7176426', '7176331'] 2023-02-17T05:39:58.340 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2023-02-17T05:39:58.340 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-small-objects-overwrites}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}'] 2023-02-17T05:39:58.341 INFO:scrape: 2023-02-17T05:39:58.341 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2023-02-17T05:39:58.341 INFO:scrape:1 jobs: ['7176396'] 2023-02-17T05:39:58.342 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect_promote_tests}'] 2023-02-17T05:39:58.342 INFO:scrape: 2023-02-17T05:39:58.342 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exceptions=0\'' 2023-02-17T05:39:58.342 INFO:scrape:1 jobs: ['7176419'] 2023-02-17T05:39:58.343 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{rhel_8}}'] 2023-02-17T05:39:58.343 INFO:scrape: 2023-02-17T05:39:58.343 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2023-02-17T05:39:58.344 INFO:scrape:1 jobs: ['7176378'] 2023-02-17T05:39:58.344 INFO:scrape:suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/quincy', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2023-02-17T05:39:58.344 INFO:scrape: 2023-02-17T05:39:58.344 INFO:scrape:Failure: {'smithi188.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman-docker-3:4.3.1-2.module_el8.8.0+1254+78119b6e.noarch: Cannot download, all mirrors were already tried without success', 'results': []}, 'smithi172.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman-catatonit-3:4.3.1-2.module_el8.8.0+1254+78119b6e.x86_64: Cannot download, all mirrors were already tried without success', 'results': []}, 'smithi005.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman-docker-3:4.3.1-2.module_el8.8.0+1254+78119b6e.noarch: Cannot download, all mirrors were already tried without success', 'results': []}, 'smithi072.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['podman', 'podman-docker'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: podman-catatonit-3:4.3.1-2.module_el8.8.0+1254+78119b6e.x86_64: Cannot download, all mirrors were already tried without success', 'results': []}} 2023-02-17T05:39:58.345 INFO:scrape:1 jobs: ['7176327'] 2023-02-17T05:39:58.345 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2023-02-17T05:39:58.345 INFO:scrape: