2021-03-08T09:36:56.911 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-08T09:36:56.915 INFO:teuthology.results:seed: '2477' 2021-03-08T09:37:39.613 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-08T09:37:39.618 INFO:teuthology.results:Waiting up to 7200 seconds for tests to finish... 2021-03-08T10:01:41.422 INFO:teuthology.results:Tests finished! gathering results... 2021-03-08T10:01:41.629 INFO:scrape:Found 1 jobs 2021-03-08T10:01:41.804 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1646-g1cbcfeff (1cbcfeffb31882cc49baf3f06acedb4bb72a3ebc) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7fce5077b638] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x560fff894552] 2021-03-08T10:01:41.806 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x8000: txn(seq 9 len 0x56 crc 0x167e249) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x8000: op_file_update file(ino 8 size 0x0 mtime 2021-03-08T09:53:40.232055+0000 allocated 0 extents []) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x8000: op_dir_link db/000004.sst to 8 2021-03-08T10:01:41.807 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_dir_unlink db/OPTIONS-000010.dbtmp , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_dir_unlink db/OPTIONS-000005 , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_file_remove 7 2021-03-08T10:01:41.809 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x29000: txn(seq 42 len 0x24 crc 0x50f08ff2) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x29000: op_file_update file(ino 16 size 0x8c4d mtime 2021-03-08T09:54:02.071826+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x3d000: txn(seq 62 len 0x24 crc 0x683cf166) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x3d000: op_file_update file(ino 16 size 0xc9dc mtime 2021-03-08T09:54:03.075552+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x3e000~1000 from file(ino 1 size 0x3e000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.811 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x51000: op_file_update file(ino 16 size 0x10f69 mtime 2021-03-08T09:54:03.105950+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x52000~1000 from file(ino 1 size 0x52000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _read left 0xae000 len 0x1000 2021-03-08T10:01:41.812 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x65000: op_dir_link db/000010.sst to 18 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x65000: op_file_update file(ino 18 size 0x5557 mtime 2021-03-08T09:54:12.769670+0000 allocated 10000 extents [1:0x4b0000~10000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x66000~1000 from file(ino 1 size 0x66000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.814 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_dir_unlink db/OPTIONS-000016.dbtmp , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_dir_unlink db/OPTIONS-000011 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_file_remove 17 2021-03-08T10:01:41.815 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:41.816 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:41.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs open_for_read db/000018.sst (random) , 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs open_for_read h 0x7f2718004f20 on file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs _read h 0x7f2718004f20 0x35b3~1000 from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) prefetch 2021-03-08T10:01:41.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment kv_backend = rocksdb , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:41.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.029+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluefs _replay 0xb000: txn(seq 12 len 0x4c crc 0x3752ac14) , 2021-03-08T09:54:16.029+0000 7f27480dadc0 20 bluefs _replay 0xb000: op_dir_unlink db/CURRENT 2021-03-08T10:01:41.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read left 0xe7000 len 0x1000 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x19000: txn(seq 26 len 0x24 crc 0xac68b3d8) 2021-03-08T10:01:41.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x2d000: txn(seq 46 len 0x24 crc 0x15e3ab73) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x2d000: op_file_update file(ino 16 size 0x8ce5 mtime 2021-03-08T09:54:02.078862+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.823 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x41000: txn(seq 66 len 0x24 crc 0x5306814b) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x41000: op_file_update file(ino 16 size 0xd80a mtime 2021-03-08T09:54:03.076341+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _read h 0x560fa7686bf0 0x42000~1000 from file(ino 1 size 0x42000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.824 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x55000: op_file_update file(ino 16 size 0x1257d mtime 2021-03-08T09:54:11.077559+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _read h 0x560fa7686bf0 0x56000~1000 from file(ino 1 size 0x56000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read left 0xaa000 len 0x1000 2021-03-08T10:01:41.825 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_dir_unlink db/CURRENT , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_file_remove 15 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_dir_link db/CURRENT to 20 2021-03-08T10:01:41.826 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_file_remove 13 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_dir_unlink db/000004.sst , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_file_remove 8 2021-03-08T10:01:41.827 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.manifest_preallocation_size: 4194304 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.is_fd_close_on_exec: 1 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.advise_random_on_open: 1 2021-03-08T10:01:41.829 WARNING:scrape:Ignoring malparsed backtrace: pin_top_level_index_and_filter: 1 , index_type: 0 , data_block_index_type: 0 2021-03-08T10:01:41.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read reaching (or past) eof, len clipped to 0x1bf , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read fetching 0x0~1000 of 1:0x450000~10000 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read left 0x1000 len 0x1bf 2021-03-08T10:01:41.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.032+0000 7f27480dadc0 10 wait_for_aio 0x560fa75d9810 , 2021-03-08T09:54:16.032+0000 7f27480dadc0 10 bluefs wait_for_aio 0x560fa75d9810 done in 0.000018 , 2021-03-08T09:54:16.032+0000 7f27480dadc0 20 bluefs flush_bdev 2021-03-08T10:01:41.832 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.033+0000 7f27480dadc0 20 bluefs _flush_and_sync_log cleaned file file(ino 32 size 0x14ad mtime 2021-03-08T09:54:16.034455+0000 allocated 10000 extents [1:0x450000~10000]) , 2021-03-08T09:54:16.033+0000 7f27480dadc0 10 bluefs _should_compact_log current 0x7a000 expected 3000 ratio 40.6667 , 2021-03-08T09:54:16.033+0000 7f27480dadc0 10 bluefs _flush 0x560fa75d5510 ignoring, length 0 < min_flush_size 524288 2021-03-08T10:01:41.833 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.034+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach 2.7_head fresh osr 0x560fa76a8090 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta 0x560fa768a3a0 cnode(bits 0) , 2021-03-08T09:54:16.034+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach meta fresh osr 0x560fa79e82d0 2021-03-08T10:01:41.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.3_head 0x560fa75c9c60) r 0 v.len 29 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore.OnodeSpace(0x560fa75c9da0 in 0x560fa7679380) add #2:c0000000::::head# 0x560fa7698250 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 _add 0x560fa7679380 #2:c0000000::::head# added, num=6 2021-03-08T10:01:41.836 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27167fc700 20 bluefs _flush_range h 0x560fa75d9810 pos now 0x67 , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 wait_for_aio 0x560fa75d9810 2021-03-08T10:01:41.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finalize_kv txc 0x560fa79e8a20 allocated 0x[] released 0x[] , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _flush 0x560fa7680010 0x7b000~1000 to file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _flush_range 0x560fa7680010 pos 0x7b000 0x7b000~1000 to file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all , 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all osr_set 0x560fa75e8270,0x560fa7665f10,0x560fa76662c0,0x560fa7692ba0,0x560fa7692d80,0x560fa76945a0,0x560fa76a8090,0x560fa79e82d0 , 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) deferred_try_submit 0 osrs, 0 txcs 2021-03-08T10:01:41.840 WARNING:scrape:Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/rados.py", line 277, in task 2021-03-08T10:01:41.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.635+0000 7f282b7fe700 10 -- >> v2:172.21.15.46:3300/0 conn(0x563578db1400 msgr2=0x563578db37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 , 2021-03-08T09:54:16.635+0000 7f282bfff700 20 --1- >> v1:172.21.15.46:6789/0 conn(0x563578dacfb0 0x563578dad3f0 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write r=0 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 -- >> v2:172.21.15.46:3300/0 conn(0x563578db1400 msgr2=0x563578db37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2021-03-08T10:01:41.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 10 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=6 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done payload.length()=402 2021-03-08T10:01:41.844 WARNING:scrape:Ignoring malparsed backtrace: epoch 1 , fsid 4ab3c265-2973-4ab7-aa95-f86b36edad73 , last_changed 2021-03-08T09:53:36.414452+0000 2021-03-08T10:01:41.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.636+0000 7f284002fdc0 5 --2- 172.21.15.46:0/3256492216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563578db1400 0x563578db37f0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state , 2021-03-08T09:54:16.636+0000 7f284002fdc0 20 Event(0x563578d7c920 nevent=5000 time_id=3).wakeup , 2021-03-08T09:54:16.636+0000 7f284002fdc0 10 --2- 172.21.15.46:0/3256492216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563578db1400 0x563578db37f0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started 2021-03-08T10:01:41.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _replay 0x4000: txn(seq 5 len 0x31 crc 0x6c40b5c0) 2021-03-08T10:01:41.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _replay 0x12000: op_dir_unlink db/CURRENT 2021-03-08T10:01:41.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _read h 0x563578db17e0 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xdc000 len 0x1000 2021-03-08T10:01:41.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _read h 0x563578db17e0 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xc8000 len 0x1000 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 2021-03-08T10:01:41.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x4c000: txn(seq 77 len 0x24 crc 0xae9c2ac6) 2021-03-08T10:01:41.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x60000: op_file_update file(ino 16 size 0x12745 mtime 2021-03-08T09:54:11.086495+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:41.855 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x6f000: op_dir_unlink db/CURRENT 2021-03-08T10:01:41.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x7b000: op_file_update file(ino 31 size 0x14d mtime 2021-03-08T09:54:16.036261+0000 allocated 10000 extents [1:0x460000~10000]) 2021-03-08T10:01:41.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.advise_random_on_open: 1 , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.db_write_buffer_size: 0 , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.write_buffer_manager: 0x563578de22f0 2021-03-08T10:01:41.859 WARNING:scrape:Ignoring malparsed backtrace: block_restart_interval: 16 , index_block_restart_interval: 1 , metadata_block_size: 4096 2021-03-08T10:01:41.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: [version_set.cc:4558] Recovered from manifest file:db/MANIFEST-000020 succeeded,manifest_file_number is 20, next_file_number is 22, last_sequence is 222, log_number is 16,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 , , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: [version_set.cc:4574] Column family [default] (ID 0), log number is 16 2021-03-08T10:01:41.862 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 10 bluefs _open_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _init_alloc , 2021-03-08T09:54:16.902+0000 7f284002fdc0 1 bluefs _init_alloc shared, id 1, capacity 0x1680000000, block size 0x10000 2021-03-08T10:01:41.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_file_remove 6 , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_dir_unlink db/MANIFEST-000001 , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_file_remove 4 2021-03-08T10:01:41.864 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0x1b000: op_file_update file(ino 16 size 0x51c8 mtime 2021-03-08T09:53:59.056543+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x1c000~1000 from file(ino 1 size 0x1c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _read left 0xe4000 len 0x1000 2021-03-08T10:01:41.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x30000~1000 from file(ino 1 size 0x30000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0xd0000 len 0x1000 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 2021-03-08T10:01:41.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0xbc000 len 0x1000 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x44000: txn(seq 69 len 0x24 crc 0xc5278ae2) 2021-03-08T10:01:41.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x58000: txn(seq 89 len 0x30 crc 0x6c84aec5) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x58000: op_file_update file(ino 16 size 0x12615 mtime 2021-03-08T09:54:11.077952+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:41.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x69000: op_file_remove 14 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x6a000~1000 from file(ino 1 size 0x6a000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0x96000 len 0x1000 2021-03-08T10:01:41.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x76000: txn(seq 119 len 0x57 crc 0x50b5bec4) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x76000: op_file_update file(ino 30 size 0x0 mtime 2021-03-08T09:54:16.033729+0000 allocated 0 extents []) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x76000: op_dir_link db/000020.dbtmp to 30 2021-03-08T10:01:41.872 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: DB SUMMARY , , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs readdir db 2021-03-08T10:01:41.873 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: kSnappyCompression supported: 1 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Fast CRC32 supported: Supported on x86 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: [db_impl/db_impl_readonly.cc:22] Opening the db in read only mode 2021-03-08T10:01:41.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-08T10:01:41.875 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.904+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta 0x563578e013b0 cnode(bits 0) , 2021-03-08T09:54:16.904+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach meta fresh osr 0x563578e01580 , 2021-03-08T09:54:16.904+0000 7f284002fdc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_on_open <<>> check (regular) start 2021-03-08T10:01:41.877 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.908+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #-1:8bb276bf:::osdmap.7:0# , 2021-03-08T09:54:16.908+0000 7f284002fdc0 20 bluestore.blob(0x563578de2740) get_ref 0x0~538 Blob(0x563578de2740 blob([0x9000~1000] csum crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x563578de77b0 sbid 0x0)) , 2021-03-08T09:54:16.908+0000 7f284002fdc0 20 bluestore.blob(0x563578de2740) get_ref init 0x1000, 1000 2021-03-08T10:01:41.878 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578c8a110 in 0x563578d16140) clear 0 , 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578e00d90 in 0x563578d16140) clear 0 , 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578c7cda0 in 0x563578d16140) clear 0 2021-03-08T10:01:41.879 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_dir_link db/OPTIONS-000005 to 7 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_dir_unlink db/OPTIONS-000004.dbtmp , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_file_update file(ino 6 size 0x133 mtime 2021-03-08T09:53:37.920382+0000 allocated 10000 extents [1:0x450000~10000]) 2021-03-08T10:01:41.880 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x14ad mtime 2021-03-08T09:53:55.321689+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x15000~1000 from file(ino 1 size 0x15000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0xeb000 len 0x1000 2021-03-08T10:01:41.882 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0xd8000 len 0x1000 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x28000: txn(seq 41 len 0x24 crc 0x1b12e88d) 2021-03-08T10:01:41.883 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x3c000: txn(seq 61 len 0x24 crc 0xf599ecfa) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x3c000: op_file_update file(ino 16 size 0xc4d6 mtime 2021-03-08T09:54:03.075304+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.884 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x50000: txn(seq 81 len 0x30 crc 0x65b0d84) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x50000: op_file_update file(ino 16 size 0x10b5d mtime 2021-03-08T09:54:03.105708+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x51000~1000 from file(ino 1 size 0x51000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.885 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x64000: op_file_update file(ino 16 size 0x127f0 mtime 2021-03-08T09:54:11.122485+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x65000~1000 from file(ino 1 size 0x65000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0x9b000 len 0x1000 2021-03-08T10:01:41.886 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x14ad mtime 2021-03-08T09:54:14.368641+0000 allocated 10000 extents [1:0x490000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x72000~1000 from file(ino 1 size 0x72000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0x8e000 len 0x1000 2021-03-08T10:01:41.888 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs stat dir sharding not found , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (sequential) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578df2fb0 on file(ino 30 size 0x10 mtime 2021-03-08T09:54:16.033765+0000 allocated 10000 extents [1:0x430000~10000]) 2021-03-08T10:01:41.889 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.two_write_queues: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.manual_wal_flush: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.atomic_flush: 0 2021-03-08T10:01:41.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-08T10:01:41.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578de6380 on file(ino 31 size 0x17f mtime 2021-03-08T09:54:16.036433+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:17.404+0000 7f284002fdc0 10 bluefs _read h 0x563578de6380 0x0~8000 from file(ino 31 size 0x17f mtime 2021-03-08T09:54:16.036433+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:17.404+0000 7f284002fdc0 20 bluefs _read reaching (or past) eof, len clipped to 0x17f 2021-03-08T10:01:41.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0x1000: op_file_update file(ino 3 size 0x25 mtime 2021-03-08T09:53:37.917565+0000 allocated 10000 extents [1:0x410000~10000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x2000~1000 from file(ino 1 size 0x2000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read left 0xfe000 len 0x1000 2021-03-08T10:01:41.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0xf000: op_dir_link db/000007.sst to 13 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0xf000: op_file_update file(ino 13 size 0x4df mtime 2021-03-08T09:53:55.320697+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x10000~1000 from file(ino 1 size 0x10000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read left 0xe0000 len 0x1000 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _replay 0x20000: txn(seq 33 len 0x24 crc 0xe25a7e16) 2021-03-08T10:01:41.895 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _replay 0x34000: txn(seq 53 len 0x24 crc 0xe69616ee) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0x34000: op_file_update file(ino 16 size 0xa7f5 mtime 2021-03-08T09:54:03.069264+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _replay 0x48000: txn(seq 73 len 0x24 crc 0xeb96782) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x48000: op_file_update file(ino 16 size 0xe14b mtime 2021-03-08T09:54:03.083414+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x49000~1000 from file(ino 1 size 0x49000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.897 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x5c000: op_file_update file(ino 16 size 0x126ad mtime 2021-03-08T09:54:11.081077+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x5d000~1000 from file(ino 1 size 0x5d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _read left 0xa3000 len 0x1000 2021-03-08T10:01:41.899 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x6c000: op_dir_link db/000013.sst to 23 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x6c000: op_file_update file(ino 23 size 0x407 mtime 2021-03-08T09:54:14.367323+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x6d000~1000 from file(ino 1 size 0x6d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.899 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x78000: op_file_remove 24 , 2021-03-08T09:54:18.129 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pgid 2.3 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x79000~1000 from file(ino 1 size 0x79000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.900 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.create_if_missing: 0 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.paranoid_checks: 1 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.env: 0x563578de3120 2021-03-08T10:01:41.901 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (random) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578deeef0 on file(ino 30 size 0x10 mtime 2021-03-08T09:54:16.033765+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (sequential) 2021-03-08T10:01:41.902 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-08T10:01:41.903 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.671+0000 7f284002fdc0 10 bluefs _flush 0x563578c923f0 0x7d000~1000 to file(ino 1 size 0x7d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.671+0000 7f284002fdc0 10 bluefs _flush_range 0x563578c923f0 pos 0x7d000 0x7d000~1000 to file(ino 1 size 0x7d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.671+0000 7f284002fdc0 20 bluefs _flush_range file now file(ino 1 size 0x7e000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.904 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _flush_range h 0x563578c923f0 pos now 0x80000 , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 wait_for_aio 0x563578c923f0 2021-03-08T10:01:41.905 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _read left 0x2000 len 0x14ad , 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _read got 5293 , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 bluefs rename db/OPTIONS-000025.dbtmp -> db/OPTIONS-000026 2021-03-08T10:01:41.906 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread start , 2021-03-08T09:54:17.673+0000 7f27fcff9700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread sleep , 2021-03-08T09:54:17.673+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _deferred_replay start 2021-03-08T10:01:41.907 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f284002fdc0 15 bluestore(/var/lib/ceph/osd/ceph-0) omap_get_values 2.0_head oid #2:00000000::::head# , 2021-03-08T09:54:17.673+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.0_head 0x563578c35810) get_onode oid #2:00000000::::head# key 0x7F800000000000000200000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F , 2021-03-08T09:54:17.673+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.0_head 0x563578c35810) r 0 v.len 29 2021-03-08T10:01:41.908 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 wait_for_aio 0x563578dc8010 , 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluefs wait_for_aio 0x563578dc8010 done in 0.000021 2021-03-08T10:01:41.908 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.674+0000 7f281a7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_committed_kv txc 0x563578dc90f0 , 2021-03-08T09:54:17.674+0000 7f281a7fc700 10 bluestore(/var/lib/ceph/osd/ceph-0) _txc_state_proc txc 0x563578dc90f0 finishing , 2021-03-08T09:54:17.674+0000 7f281a7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish 0x563578dc90f0 onodes 0x563578dc9910 2021-03-08T10:01:41.909 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.674+0000 7f284002fdc0 20 bluefs _stop_alloc , 2021-03-08T09:54:17.674+0000 7f284002fdc0 1 bdev(0x563578d19b30 /var/lib/ceph/osd/ceph-0/block) close , 2021-03-08T09:54:17.939+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _close_fm 2021-03-08T10:01:41.910 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 10 bluefs _replay 0x7000: txn(seq 8 len 0x63 crc 0x113625d1) , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x7000: op_dir_link db/OPTIONS-000005 to 7 2021-03-08T10:01:41.910 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x0 mtime 2021-03-08T09:53:55.321586+0000 allocated 0 extents []) , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_dir_link db/OPTIONS-000010.dbtmp to 17 , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x14ad mtime 2021-03-08T09:53:55.321689+0000 allocated 10000 extents [1:0x470000~10000]) 2021-03-08T10:01:41.911 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _replay 0x27000: op_file_update file(ino 16 size 0x8c01 mtime 2021-03-08T09:54:02.070915+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x28000~1000 from file(ino 1 size 0x28000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xd8000 len 0x1000 2021-03-08T10:01:41.912 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x3c000~1000 from file(ino 1 size 0x3c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xc4000 len 0x1000 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.912 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xb0000 len 0x1000 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _replay 0x50000: txn(seq 81 len 0x30 crc 0x65b0d84) 2021-03-08T10:01:41.913 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _replay 0x64000: txn(seq 101 len 0x30 crc 0x7f93623a) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _replay 0x64000: op_file_update file(ino 16 size 0x127f0 mtime 2021-03-08T09:54:11.122485+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:41.914 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x0 mtime 2021-03-08T09:54:14.368528+0000 allocated 0 extents []) , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_dir_link db/OPTIONS-000016.dbtmp to 27 , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x14ad mtime 2021-03-08T09:54:14.368641+0000 allocated 10000 extents [1:0x490000~10000]) 2021-03-08T10:01:41.914 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x7e000: op_dir_link db/MANIFEST-000023 to 34 , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x7e000: op_file_update file(ino 34 size 0xd3 mtime 2021-03-08T09:54:17.672825+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x7f000~1000 from file(ino 1 size 0x7f000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.915 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 10 bluefs stat db/MANIFEST-000023 file(ino 34 size 0xd3 mtime 2021-03-08T09:54:17.672825+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: MANIFEST file: MANIFEST-000023 size: 211 Bytes , 2021-03-08T10:01:41.916 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x10 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 20 bluefs _read got 16 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54d160 0x10~2000 from file(ino 35 size 0x10 mtime 2021-03-08T09:54:17.672989+0000 allocated 10000 extents [1:0x4b0000~10000]) 2021-03-08T10:01:41.916 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-08T10:01:41.917 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 20 bluefs _read_random left 0xe39 0x41c7~8c , 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 10 bluefs _read_random h 0x7fccf8001d20 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) 2021-03-08T10:01:41.918 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 20 bluefs _replay 0x3000: op_file_update file(ino 5 size 0x10 mtime 2021-03-08T09:53:37.917928+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 20 bluefs _read left 0xfc000 len 0x1000 2021-03-08T10:01:41.918 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x12000~1000 from file(ino 1 size 0x12000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.919 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.920 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _replay 0x37000: txn(seq 56 len 0x24 crc 0x2df8dbf5) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.920 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x4b000: op_file_update file(ino 16 size 0xf629 mtime 2021-03-08T09:54:03.086479+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xb4000 len 0x1000 2021-03-08T10:01:41.921 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x60000~1000 from file(ino 1 size 0x60000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.922 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x6f000~1000 from file(ino 1 size 0x6f000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.922 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x7b000~1000 from file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.923 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a557940 0x10~2000 from file(ino 35 size 0x10 mtime 2021-03-08T09:54:17.672989+0000 allocated 10000 extents [1:0x4b0000~10000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 0 2021-03-08T10:01:41.924 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.max_background_jobs: 2 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-08T10:01:41.924 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.level: 32767 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-08T10:01:41.925 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x25 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~1000 of 1:0x410000~10000 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x25 2021-03-08T10:01:41.926 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluestore.BufferSpace(0x55fb4a55c5e8 in 0x55fb4a4a5a10) _clear , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection meta cnode(bits 0) , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #-1:3e70e826:::osdmap.12:0# 2021-03-08T10:01:41.927 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluefs _read_random got 3791 , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_on_open checking for stray omap data 2021-03-08T10:01:41.927 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.650+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x0~1000 from file(ino 1 size 0x1000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.650+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~100000 of 1:0x10000~400000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0x100000 len 0x1000 2021-03-08T10:01:41.928 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0xd000: txn(seq 14 len 0x60 crc 0xe012ceaf) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0xd000: op_file_update file(ino 12 size 0x0 mtime 2021-03-08T09:53:40.233346+0000 allocated 0 extents []) 2021-03-08T10:01:41.928 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x1c000: op_file_update file(ino 16 size 0x51ee mtime 2021-03-08T09:53:59.056942+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x1d000~1000 from file(ino 1 size 0x1d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xe3000 len 0x1000 2021-03-08T10:01:41.929 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x31000~1000 from file(ino 1 size 0x31000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xcf000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.930 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xbb000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x45000: txn(seq 70 len 0x24 crc 0x51520b6e) 2021-03-08T10:01:41.930 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x59000: txn(seq 90 len 0x30 crc 0x2ec16f6b) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x59000: op_file_update file(ino 16 size 0x1263b mtime 2021-03-08T09:54:11.078081+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:41.931 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_file_update file(ino 22 size 0x0 mtime 2021-03-08T09:54:12.770693+0000 allocated 0 extents []) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_dir_link db/OPTIONS-000013.dbtmp to 22 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_file_update file(ino 22 size 0x14ad mtime 2021-03-08T09:54:12.770777+0000 allocated 10000 extents [1:0x4e0000~10000]) 2021-03-08T10:01:41.932 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0x89000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x77000: txn(seq 120 len 0x4c crc 0xb16f6660) 2021-03-08T10:01:41.932 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 10 bluefs _replay 0x84000: txn(seq 133 len 0x23 crc 0xb910e119) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 20 bluefs _replay 0x84000: op_file_update file(ino 36 size 0x61f mtime 2021-03-08T09:54:17.675013+0000 allocated 10000 extents [1:0x4d0000~10000]) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x85000~1000 from file(ino 1 size 0x85000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.933 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.use_adaptive_mutex: 0 , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.rate_limiter: (nil) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-08T10:01:41.934 WARNING:scrape:Ignoring malparsed backtrace: whole_key_filtering: 1 , verify_compression: 0 , read_amp_bytes_per_bit: 0 2021-03-08T10:01:41.934 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 10 bluefs _read_random h 0x7fcd04005050 0x188~2b8 from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 20 bluefs _read_random left 0xe78 0x188~2b8 , 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 20 bluefs _read_random got 696 2021-03-08T10:01:41.935 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 10 bluefs _close_writer 0x55fb4a49c4d0 type 0 , 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 20 bluefs _stop_alloc , 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 1 bdev(0x55fb4a542a80 /var/lib/ceph/osd/ceph-0/block) close 2021-03-08T10:01:41.936 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 20 bluefs _replay 0xa000: op_dir_link db/000005.dbtmp to 10 , 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.936 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.937 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _replay 0x2c000: txn(seq 45 len 0x24 crc 0x5f1d3bb0) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.938 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x40000: op_file_update file(ino 16 size 0xd7e4 mtime 2021-03-08T09:54:03.076199+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0xbf000 len 0x1000 2021-03-08T10:01:41.938 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x55000~1000 from file(ino 1 size 0x55000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.939 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x68000~1000 from file(ino 1 size 0x68000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.939 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_file_remove 23 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_file_remove 18 2021-03-08T10:01:41.940 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_file_remove 31 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_file_remove 29 2021-03-08T10:01:41.941 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.info_log: 0x55fb4a47f510 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.max_file_opening_threads: 16 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.statistics: (nil) 2021-03-08T10:01:41.942 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x10 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~1000 of 1:0x4b0000~10000 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x10 2021-03-08T10:01:41.942 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-08T10:01:41.943 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _fsync 0x55fb4a4a43f0 file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _flush 0x55fb4a4a43f0 0x0~93d to file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _flush_range 0x55fb4a4a43f0 pos 0x0 0x0~93d to file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) 2021-03-08T10:01:41.944 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 10 HybridAllocator allocate want 0x10000 unit 0x10000 max_alloc_size 0x10000 hint 0x0 , 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 20 bluefs _flush_range dirty_seq = 136 (was clean) , 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 20 bluefs _flush_range file now file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) 2021-03-08T10:01:41.944 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range file now file(ino 1 size 0x8b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range in 1:0x10000~400000 x_off 0x8a000 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range h 0x55fb4a49c4d0 pos now 0x8b000 2021-03-08T10:01:41.945 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 10 bluefs _read_random h 0x7fccf8004c80 0x0~107b from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _read_random read random 0x0~107b of 1:0x4a0000~10000 2021-03-08T10:01:41.946 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd0e7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_finalize_thread sleep , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55fb4a56ce60 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all done 2021-03-08T10:01:41.946 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment do_bluefs = 1 , 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment initializing bluefs , 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block 0 2021-03-08T10:01:41.947 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_link db/CURRENT to 10 , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_unlink db/000005.dbtmp , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0xc000~1000 from file(ino 1 size 0xc000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.947 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.490+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x1a000~1000 from file(ino 1 size 0x1a000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _read left 0xe6000 len 0x1000 , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.948 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0xd2000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x2e000: txn(seq 47 len 0x24 crc 0xbc743dc4) 2021-03-08T10:01:41.949 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x42000: txn(seq 67 len 0x24 crc 0x851bc9b8) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x42000: op_file_update file(ino 16 size 0xd830 mtime 2021-03-08T09:54:03.077262+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.950 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x56000: txn(seq 87 len 0x30 crc 0xb9858be4) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x56000: op_file_update file(ino 16 size 0x125a3 mtime 2021-03-08T09:54:11.077692+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x57000~1000 from file(ino 1 size 0x57000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.950 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x69000~1000 from file(ino 1 size 0x69000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0x97000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.951 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0x8b000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x75000: txn(seq 118 len 0x5b crc 0xf271c68f) 2021-03-08T10:01:41.952 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_dir_link db/OPTIONS-000025.dbtmp to 37 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_file_update file(ino 37 size 0x14ad mtime 2021-03-08T09:54:17.673483+0000 allocated 10000 extents [1:0x4c0000~10000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x83000~1000 from file(ino 1 size 0x83000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.952 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 20 bluefs _read got 279 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 5 rocksdb: verify_sharding column families from rocksdb: [default] , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 10 bluefs open_for_read sharding/recreate_columns (sequential) 2021-03-08T10:01:41.953 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.stats_history_buffer_size: 1048576 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.max_open_files: -1 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.bytes_per_sync: 0 2021-03-08T10:01:41.953 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.target_file_size_multiplier: 1 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2021-03-08T10:01:41.954 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.493+0000 7ff6167fc700 10 bluefs _read_random h 0x7ff600001d20 0x11a~4a from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:20.493+0000 7ff6167fc700 20 bluefs _read_random left 0xee6 0x11a~4a , 2021-03-08T09:54:20.493+0000 7ff6167fc700 20 bluefs _read_random got 74 2021-03-08T10:01:41.955 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment initializing bluefs , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block 0 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 1 bdev(0x5646fca28a30 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2021-03-08T10:01:41.955 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_unlink db/000005.dbtmp , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0xc000~1000 from file(ino 1 size 0xc000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read left 0xf4000 len 0x1000 2021-03-08T10:01:41.956 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read left 0xe6000 len 0x1000 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs _replay 0x1a000: txn(seq 27 len 0x24 crc 0x91ca6867) 2021-03-08T10:01:41.957 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x2e000: txn(seq 47 len 0x24 crc 0xbc743dc4) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x2e000: op_file_update file(ino 16 size 0x9e52 mtime 2021-03-08T09:54:03.064379+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:41.957 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x42000: txn(seq 67 len 0x24 crc 0x851bc9b8) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x42000: op_file_update file(ino 16 size 0xd830 mtime 2021-03-08T09:54:03.077262+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x43000~1000 from file(ino 1 size 0x43000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.958 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x56000: op_file_update file(ino 16 size 0x125a3 mtime 2021-03-08T09:54:11.077692+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x57000~1000 from file(ino 1 size 0x57000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0xa9000 len 0x1000 2021-03-08T10:01:41.959 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0x97000 len 0x1000 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x69000: txn(seq 106 len 0x7c crc 0x67e69dad) 2021-03-08T10:01:41.959 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x75000: txn(seq 118 len 0x5b crc 0xf271c68f) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x75000: op_file_update file(ino 29 size 0x0 mtime 2021-03-08T09:54:16.033410+0000 allocated 0 extents []) 2021-03-08T10:01:41.960 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_file_update file(ino 37 size 0x14ad mtime 2021-03-08T09:54:17.673483+0000 allocated 10000 extents [1:0x4c0000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x83000~1000 from file(ino 1 size 0x83000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0x7d000 len 0x1000 2021-03-08T10:01:41.961 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 20 bluefs _read got 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 10 bluefs open_for_read db/MANIFEST-000026 (sequential) 2021-03-08T10:01:41.961 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:41.962 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-08T10:01:41.963 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 10 bluefs _read_random h 0x7ff600003ab0 0x58c~4a from file(ino 38 size 0x93d mtime 2021-03-08T09:54:19.917885+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 20 bluefs _read_random left 0xa74 0x58c~4a , 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 20 bluefs _read_random got 74 2021-03-08T10:01:41.963 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow 0x0~a59: 0x0~a59 Blob(0x5646fca2c190 blob([0xf000~1000] csum crc32c/0x1000) use_tracker(0x1000 0xa59) SharedBlob(0x5646fcac6970 sbid 0x0)) , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore.sharedblob(0x5646fcac6970) put 0x5646fcac6970 removing self from set 0x5646fcac6570 , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore.BufferSpace(0x5646fcac6988 in 0x5646fc98ba10) _clear 2021-03-08T10:01:41.964 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection 1.0_head cnode(bits 0) , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:00000000::::head# , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:d4465517:::INTEL_SSDPED1D480GA_PHMB7512009A480DGN:head# 2021-03-08T10:01:41.965 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.254+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:21.254+0000 7ff636d71dc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:21.254+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:41.965 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:41.966 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.967 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:41.967 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.968 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:41.969 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:41.969 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:41.970 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.971 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 20 bluefs _read got 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 10 bluefs open_for_read db/MANIFEST-000026 (sequential) , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 10 bluefs open_for_read h 0x5646fca12750 on file(ino 39 size 0x117 mtime 2021-03-08T09:54:19.918263+0000 allocated 10000 extents [1:0x430000~10000]) 2021-03-08T10:01:41.971 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-08T10:01:41.972 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.enabled: false 2021-03-08T10:01:41.972 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff6167fc700 20 bluefs _read_random got 85 , 2021-03-08T09:54:21.256+0000 7ff6167fc700 10 bluefs _read_random h 0x7ff604004aa0 0x188~2b8 from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:21.256+0000 7ff6167fc700 20 bluefs _read_random left 0xe78 0x188~2b8 2021-03-08T10:01:41.973 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.257+0000 7ff636d71dc0 1 bdev(0x5646fca28a30 /var/lib/ceph/osd/ceph-0/block) close , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) 2021-03-08T10:01:41.974 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xf5000 len 0x1000 , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.974 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xe7000 len 0x1000 2021-03-08T10:01:41.975 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xd3000 len 0x1000 , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:41.976 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x41000: txn(seq 66 len 0x24 crc 0x5306814b) 2021-03-08T10:01:41.976 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x55000: op_file_update file(ino 16 size 0x1257d mtime 2021-03-08T09:54:11.077559+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:41.977 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x68000: op_dir_unlink db/CURRENT 2021-03-08T10:01:41.978 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 13 2021-03-08T10:01:41.978 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read left 0x7e000 len 0x1000 2021-03-08T10:01:41.979 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs open_for_read h 0x5646fca0e110 on file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca0e110 0x0~2000 from file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x10 2021-03-08T10:01:41.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.atomic_flush: 0 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-08T10:01:41.980 WARNING:scrape:Ignoring malparsed backtrace: verify_compression: 0 , read_amp_bytes_per_bit: 0 , format_version: 2 2021-03-08T10:01:41.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.520+0000 7ff5f6ffd700 20 bluefs _read_random got 697 , 2021-03-08T09:54:21.520+0000 7ff616ffd700 20 bluefs _read left 0x1000 len 0x4ca , 2021-03-08T09:54:21.520+0000 7ff616ffd700 20 bluefs _read got 1226 2021-03-08T10:01:41.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 wait_for_aio 0x5646fc9824d0 , 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 bluefs wait_for_aio 0x5646fc9824d0 done in 0.000019 2021-03-08T10:01:41.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs _flush_range in 1:0x4b0000~10000 x_off 0x0 , 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs flush_buffer caching tail of 0x4ad and padding block with 0xb53 buffer.length() 0 , 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs _flush_range h 0x5646fc980510 pos now 0x14ad 2021-03-08T10:01:41.983 WARNING:scrape:Ignoring malparsed backtrace: Flush(GB): cumulative 0.000, interval 0.000 , AddFile(GB): cumulative 0.000, interval 0.000 , AddFile(Total Files): cumulative 0, interval 0 2021-03-08T10:01:41.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.522+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x5646fca2a830 , 2021-03-08T09:54:21.522+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x5646fca32650 , 2021-03-08T09:54:21.522+0000 7ff5fd7fa700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread sleep 2021-03-08T10:01:41.984 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.631+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:22.631+0000 7fb50e40adc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:22.631+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:41.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:41.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.986 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:41.987 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:41.987 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:41.988 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:41.989 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:41.989 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.990 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _replay 0x8f000: txn(seq 144 len 0x7c crc 0x97e17074) , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x8f000: op_file_update file(ino 45 size 0x0 mtime 2021-03-08T09:54:21.522082+0000 allocated 0 extents []) 2021-03-08T10:01:41.991 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:41.991 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:41.992 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs open_for_read db/000022.sst (random) , 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs open_for_read h 0x7fb4dc001d20 on file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs _read h 0x7fb4dc001d20 0x0~4ca from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) prefetch 2021-03-08T10:01:41.993 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.635+0000 7fb50e40adc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:22.635+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4d4001d20 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:22.635+0000 7fb50e40adc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 2021-03-08T10:01:41.993 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.895+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:41.994 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) 2021-03-08T10:01:41.994 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:41.995 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xc8000 len 0x1000 2021-03-08T10:01:41.996 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:41.996 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) 2021-03-08T10:01:41.997 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) 2021-03-08T10:01:41.998 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) 2021-03-08T10:01:41.998 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x89000: txn(seq 138 len 0x7c crc 0x140649b0) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_file_update file(ino 41 size 0x0 mtime 2021-03-08T09:54:19.918678+0000 allocated 0 extents []) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_dir_link db.wal/000027.log to 41 2021-03-08T10:01:41.999 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 0 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs open_for_read db/MANIFEST-000029 (sequential) 2021-03-08T10:01:42.000 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.000 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-08T10:01:42.001 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb4f67fc700 20 bluefs _read_random left 0xa06 0x5fa~2b9 , 2021-03-08T09:54:22.897+0000 7fb4f67fc700 20 bluefs _read_random got 697 , 2021-03-08T09:54:22.897+0000 7fb4ca7fc700 10 bluefs _read_random h 0x7fb4e0003ac0 0x11a~4a from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) 2021-03-08T10:01:42.002 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow 0x0~a59: 0x0~a59 Blob(0x55eec084d190 blob([0xf000~1000] csum crc32c/0x1000) use_tracker(0x1000 0xa59) SharedBlob(0x55eec0863c00 sbid 0x0)) , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore.sharedblob(0x55eec0863c00) put 0x55eec0863c00 removing self from set 0x55eec085e460 , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore.BufferSpace(0x55eec0863c18 in 0x55eec07aca10) _clear 2021-03-08T10:01:42.002 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection 1.0_head cnode(bits 0) , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:00000000::::head# , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:d4465517:::INTEL_SSDPED1D480GA_PHMB7512009A480DGN:head# 2021-03-08T10:01:42.003 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.399+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:23.399+0000 7fb50e40adc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:23.399+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.004 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.004 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.005 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.006 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.006 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.007 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.007 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.008 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.009 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 10 bluefs _replay 0x8f000: txn(seq 144 len 0x7c crc 0x97e17074) , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 20 bluefs _replay 0x8f000: op_file_update file(ino 45 size 0x0 mtime 2021-03-08T09:54:21.522082+0000 allocated 0 extents []) 2021-03-08T10:01:42.009 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.010 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.011 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs open_for_read db/000022.sst (random) , 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs open_for_read h 0x7fb4d4004c60 on file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs _read h 0x7fb4d4004c60 0x0~4ca from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) prefetch 2021-03-08T10:01:42.011 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.402+0000 7fb50e40adc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:23.402+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4e00039c0 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:23.402+0000 7fb50e40adc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 2021-03-08T10:01:42.012 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.663+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.013 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) 2021-03-08T10:01:42.014 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.014 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xc8000 len 0x1000 2021-03-08T10:01:42.015 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.016 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) 2021-03-08T10:01:42.016 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) 2021-03-08T10:01:42.017 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) 2021-03-08T10:01:42.018 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x89000: txn(seq 138 len 0x7c crc 0x140649b0) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_file_update file(ino 41 size 0x0 mtime 2021-03-08T09:54:19.918678+0000 allocated 0 extents []) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_dir_link db.wal/000027.log to 41 2021-03-08T10:01:42.018 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 0 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs open_for_read db/MANIFEST-000029 (sequential) 2021-03-08T10:01:42.019 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.020 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.max_write_buffer_number: 4 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.compression: NoCompression , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-08T10:01:42.020 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb4f5ffb700 20 bluefs _read_random got 53 , 2021-03-08T09:54:23.665+0000 7fb4f67fc700 10 bluefs _read_random h 0x7fb4d4005cd0 0x908~35 from file(ino 38 size 0x93d mtime 2021-03-08T09:54:19.917885+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:23.665+0000 7fb4f67fc700 20 bluefs _read_random left 0x6f8 0x908~35 2021-03-08T10:01:42.021 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.666+0000 7fb50e40adc0 10 bluefs open_for_write db/000032.dbtmp , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs open_for_write mapping db/000032.dbtmp vsel_hint 0x1 , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 10 bluefs open_for_write h 0x55eec0879010 on file(ino 48 size 0x0 mtime 2021-03-08T09:54:23.667013+0000 allocated 0 extents []) 2021-03-08T10:01:42.022 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _fsync file metadata was dirty (151) on file(ino 50 size 0x14ad mtime 2021-03-08T09:54:23.667558+0000 allocated 10000 extents [1:0x490000~10000]), flushing log , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _flush_and_sync_log 1 dirty_files , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _flush_and_sync_log op_file_update file(ino 50 size 0x14ad mtime 2021-03-08T09:54:23.667558+0000 allocated 10000 extents [1:0x490000~10000]) 2021-03-08T10:01:42.022 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluefs _read_random got 282 , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4e0003ba0 0x41c7~8c from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluefs _read_random left 0xe39 0x41c7~8c 2021-03-08T10:01:42.023 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55eec085de00 , 2021-03-08T09:54:23.667+0000 7fb4f67fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_finalize_thread sleep , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55eec085e1b0 2021-03-08T10:01:42.024 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:34.490 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] , 2021-03-08T09:54:39.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2021-03-08T10:01:42.024 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:56:47.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2021-03-08T09:56:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-08T10:01:42.025 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2021-03-08T09:58:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2021-03-08T09:58:56.167 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-03-08T10:01:42.026 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/ceph_manager.py", line 173, in wrapper , return func(self) , File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/ceph_manager.py", line 1261, in _do_thrash 2021-03-08T10:01:42.027 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:00:48.076 INFO:teuthology.orchestra.run.smithi046.stdout:{ , 2021-03-08T10:00:48.077 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "" , 2021-03-08T10:00:48.077 INFO:teuthology.orchestra.run.smithi046.stdout:} 2021-03-08T10:01:42.028 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:01:10.739 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com , 2021-03-08T10:01:10.740 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2021-03-08T10:01:10.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2021-03-08T10:01:42.029 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:01:12.932 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-03-08T10:01:12.933 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com , 2021-03-08T10:01:12.933 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-03-08T10:01:42.065 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1646-g1cbcfeff (1cbcfeffb31882cc49baf3f06acedb4bb72a3ebc) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7fce5077b638] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x560fff894552] 2021-03-08T10:01:42.066 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x8000: txn(seq 9 len 0x56 crc 0x167e249) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x8000: op_file_update file(ino 8 size 0x0 mtime 2021-03-08T09:53:40.232055+0000 allocated 0 extents []) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x8000: op_dir_link db/000004.sst to 8 2021-03-08T10:01:42.067 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_dir_unlink db/OPTIONS-000010.dbtmp , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_dir_unlink db/OPTIONS-000005 , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_file_remove 7 2021-03-08T10:01:42.068 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x29000: txn(seq 42 len 0x24 crc 0x50f08ff2) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x29000: op_file_update file(ino 16 size 0x8c4d mtime 2021-03-08T09:54:02.071826+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.068 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x3d000: txn(seq 62 len 0x24 crc 0x683cf166) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x3d000: op_file_update file(ino 16 size 0xc9dc mtime 2021-03-08T09:54:03.075552+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x3e000~1000 from file(ino 1 size 0x3e000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.069 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x51000: op_file_update file(ino 16 size 0x10f69 mtime 2021-03-08T09:54:03.105950+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x52000~1000 from file(ino 1 size 0x52000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _read left 0xae000 len 0x1000 2021-03-08T10:01:42.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x65000: op_dir_link db/000010.sst to 18 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x65000: op_file_update file(ino 18 size 0x5557 mtime 2021-03-08T09:54:12.769670+0000 allocated 10000 extents [1:0x4b0000~10000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x66000~1000 from file(ino 1 size 0x66000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_dir_unlink db/OPTIONS-000016.dbtmp , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_dir_unlink db/OPTIONS-000011 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_file_remove 17 2021-03-08T10:01:42.071 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.072 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.072 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs open_for_read db/000018.sst (random) , 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs open_for_read h 0x7f2718004f20 on file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs _read h 0x7f2718004f20 0x35b3~1000 from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) prefetch 2021-03-08T10:01:42.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment kv_backend = rocksdb , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.029+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluefs _replay 0xb000: txn(seq 12 len 0x4c crc 0x3752ac14) , 2021-03-08T09:54:16.029+0000 7f27480dadc0 20 bluefs _replay 0xb000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read left 0xe7000 len 0x1000 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x19000: txn(seq 26 len 0x24 crc 0xac68b3d8) 2021-03-08T10:01:42.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x2d000: txn(seq 46 len 0x24 crc 0x15e3ab73) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x2d000: op_file_update file(ino 16 size 0x8ce5 mtime 2021-03-08T09:54:02.078862+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x41000: txn(seq 66 len 0x24 crc 0x5306814b) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x41000: op_file_update file(ino 16 size 0xd80a mtime 2021-03-08T09:54:03.076341+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _read h 0x560fa7686bf0 0x42000~1000 from file(ino 1 size 0x42000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x55000: op_file_update file(ino 16 size 0x1257d mtime 2021-03-08T09:54:11.077559+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _read h 0x560fa7686bf0 0x56000~1000 from file(ino 1 size 0x56000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read left 0xaa000 len 0x1000 2021-03-08T10:01:42.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_dir_unlink db/CURRENT , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_file_remove 15 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_dir_link db/CURRENT to 20 2021-03-08T10:01:42.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_file_remove 13 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_dir_unlink db/000004.sst , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_file_remove 8 2021-03-08T10:01:42.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.manifest_preallocation_size: 4194304 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.is_fd_close_on_exec: 1 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.advise_random_on_open: 1 2021-03-08T10:01:42.079 WARNING:scrape:Ignoring malparsed backtrace: pin_top_level_index_and_filter: 1 , index_type: 0 , data_block_index_type: 0 2021-03-08T10:01:42.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read reaching (or past) eof, len clipped to 0x1bf , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read fetching 0x0~1000 of 1:0x450000~10000 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read left 0x1000 len 0x1bf 2021-03-08T10:01:42.081 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.032+0000 7f27480dadc0 10 wait_for_aio 0x560fa75d9810 , 2021-03-08T09:54:16.032+0000 7f27480dadc0 10 bluefs wait_for_aio 0x560fa75d9810 done in 0.000018 , 2021-03-08T09:54:16.032+0000 7f27480dadc0 20 bluefs flush_bdev 2021-03-08T10:01:42.081 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.033+0000 7f27480dadc0 20 bluefs _flush_and_sync_log cleaned file file(ino 32 size 0x14ad mtime 2021-03-08T09:54:16.034455+0000 allocated 10000 extents [1:0x450000~10000]) , 2021-03-08T09:54:16.033+0000 7f27480dadc0 10 bluefs _should_compact_log current 0x7a000 expected 3000 ratio 40.6667 , 2021-03-08T09:54:16.033+0000 7f27480dadc0 10 bluefs _flush 0x560fa75d5510 ignoring, length 0 < min_flush_size 524288 2021-03-08T10:01:42.082 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.034+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach 2.7_head fresh osr 0x560fa76a8090 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta 0x560fa768a3a0 cnode(bits 0) , 2021-03-08T09:54:16.034+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach meta fresh osr 0x560fa79e82d0 2021-03-08T10:01:42.083 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.3_head 0x560fa75c9c60) r 0 v.len 29 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore.OnodeSpace(0x560fa75c9da0 in 0x560fa7679380) add #2:c0000000::::head# 0x560fa7698250 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 _add 0x560fa7679380 #2:c0000000::::head# added, num=6 2021-03-08T10:01:42.083 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27167fc700 20 bluefs _flush_range h 0x560fa75d9810 pos now 0x67 , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 wait_for_aio 0x560fa75d9810 2021-03-08T10:01:42.084 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finalize_kv txc 0x560fa79e8a20 allocated 0x[] released 0x[] , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _flush 0x560fa7680010 0x7b000~1000 to file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _flush_range 0x560fa7680010 pos 0x7b000 0x7b000~1000 to file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.084 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all , 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all osr_set 0x560fa75e8270,0x560fa7665f10,0x560fa76662c0,0x560fa7692ba0,0x560fa7692d80,0x560fa76945a0,0x560fa76a8090,0x560fa79e82d0 , 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) deferred_try_submit 0 osrs, 0 txcs 2021-03-08T10:01:42.085 WARNING:scrape:Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/rados.py", line 277, in task 2021-03-08T10:01:42.086 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.635+0000 7f282b7fe700 10 -- >> v2:172.21.15.46:3300/0 conn(0x563578db1400 msgr2=0x563578db37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 , 2021-03-08T09:54:16.635+0000 7f282bfff700 20 --1- >> v1:172.21.15.46:6789/0 conn(0x563578dacfb0 0x563578dad3f0 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write r=0 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 -- >> v2:172.21.15.46:3300/0 conn(0x563578db1400 msgr2=0x563578db37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2021-03-08T10:01:42.086 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 10 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=6 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done payload.length()=402 2021-03-08T10:01:42.087 WARNING:scrape:Ignoring malparsed backtrace: epoch 1 , fsid 4ab3c265-2973-4ab7-aa95-f86b36edad73 , last_changed 2021-03-08T09:53:36.414452+0000 2021-03-08T10:01:42.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.636+0000 7f284002fdc0 5 --2- 172.21.15.46:0/3256492216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563578db1400 0x563578db37f0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state , 2021-03-08T09:54:16.636+0000 7f284002fdc0 20 Event(0x563578d7c920 nevent=5000 time_id=3).wakeup , 2021-03-08T09:54:16.636+0000 7f284002fdc0 10 --2- 172.21.15.46:0/3256492216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563578db1400 0x563578db37f0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started 2021-03-08T10:01:42.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _replay 0x4000: txn(seq 5 len 0x31 crc 0x6c40b5c0) 2021-03-08T10:01:42.089 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _replay 0x12000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.090 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _read h 0x563578db17e0 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xdc000 len 0x1000 2021-03-08T10:01:42.090 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _read h 0x563578db17e0 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xc8000 len 0x1000 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 2021-03-08T10:01:42.091 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x4c000: txn(seq 77 len 0x24 crc 0xae9c2ac6) 2021-03-08T10:01:42.092 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x60000: op_file_update file(ino 16 size 0x12745 mtime 2021-03-08T09:54:11.086495+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.092 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x6f000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.093 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x7b000: op_file_update file(ino 31 size 0x14d mtime 2021-03-08T09:54:16.036261+0000 allocated 10000 extents [1:0x460000~10000]) 2021-03-08T10:01:42.094 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.advise_random_on_open: 1 , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.db_write_buffer_size: 0 , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.write_buffer_manager: 0x563578de22f0 2021-03-08T10:01:42.094 WARNING:scrape:Ignoring malparsed backtrace: block_restart_interval: 16 , index_block_restart_interval: 1 , metadata_block_size: 4096 2021-03-08T10:01:42.095 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: [version_set.cc:4558] Recovered from manifest file:db/MANIFEST-000020 succeeded,manifest_file_number is 20, next_file_number is 22, last_sequence is 222, log_number is 16,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 , , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: [version_set.cc:4574] Column family [default] (ID 0), log number is 16 2021-03-08T10:01:42.096 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 10 bluefs _open_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _init_alloc , 2021-03-08T09:54:16.902+0000 7f284002fdc0 1 bluefs _init_alloc shared, id 1, capacity 0x1680000000, block size 0x10000 2021-03-08T10:01:42.096 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_file_remove 6 , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_dir_unlink db/MANIFEST-000001 , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_file_remove 4 2021-03-08T10:01:42.097 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0x1b000: op_file_update file(ino 16 size 0x51c8 mtime 2021-03-08T09:53:59.056543+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x1c000~1000 from file(ino 1 size 0x1c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _read left 0xe4000 len 0x1000 2021-03-08T10:01:42.098 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x30000~1000 from file(ino 1 size 0x30000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0xd0000 len 0x1000 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 2021-03-08T10:01:42.098 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0xbc000 len 0x1000 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x44000: txn(seq 69 len 0x24 crc 0xc5278ae2) 2021-03-08T10:01:42.099 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x58000: txn(seq 89 len 0x30 crc 0x6c84aec5) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x58000: op_file_update file(ino 16 size 0x12615 mtime 2021-03-08T09:54:11.077952+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.099 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x69000: op_file_remove 14 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x6a000~1000 from file(ino 1 size 0x6a000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0x96000 len 0x1000 2021-03-08T10:01:42.100 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x76000: txn(seq 119 len 0x57 crc 0x50b5bec4) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x76000: op_file_update file(ino 30 size 0x0 mtime 2021-03-08T09:54:16.033729+0000 allocated 0 extents []) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x76000: op_dir_link db/000020.dbtmp to 30 2021-03-08T10:01:42.101 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: DB SUMMARY , , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs readdir db 2021-03-08T10:01:42.101 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: kSnappyCompression supported: 1 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Fast CRC32 supported: Supported on x86 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: [db_impl/db_impl_readonly.cc:22] Opening the db in read only mode 2021-03-08T10:01:42.102 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-08T10:01:42.103 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.904+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta 0x563578e013b0 cnode(bits 0) , 2021-03-08T09:54:16.904+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach meta fresh osr 0x563578e01580 , 2021-03-08T09:54:16.904+0000 7f284002fdc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_on_open <<>> check (regular) start 2021-03-08T10:01:42.103 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.908+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #-1:8bb276bf:::osdmap.7:0# , 2021-03-08T09:54:16.908+0000 7f284002fdc0 20 bluestore.blob(0x563578de2740) get_ref 0x0~538 Blob(0x563578de2740 blob([0x9000~1000] csum crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x563578de77b0 sbid 0x0)) , 2021-03-08T09:54:16.908+0000 7f284002fdc0 20 bluestore.blob(0x563578de2740) get_ref init 0x1000, 1000 2021-03-08T10:01:42.104 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578c8a110 in 0x563578d16140) clear 0 , 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578e00d90 in 0x563578d16140) clear 0 , 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578c7cda0 in 0x563578d16140) clear 0 2021-03-08T10:01:42.105 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_dir_link db/OPTIONS-000005 to 7 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_dir_unlink db/OPTIONS-000004.dbtmp , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_file_update file(ino 6 size 0x133 mtime 2021-03-08T09:53:37.920382+0000 allocated 10000 extents [1:0x450000~10000]) 2021-03-08T10:01:42.105 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x14ad mtime 2021-03-08T09:53:55.321689+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x15000~1000 from file(ino 1 size 0x15000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0xeb000 len 0x1000 2021-03-08T10:01:42.106 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0xd8000 len 0x1000 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x28000: txn(seq 41 len 0x24 crc 0x1b12e88d) 2021-03-08T10:01:42.107 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x3c000: txn(seq 61 len 0x24 crc 0xf599ecfa) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x3c000: op_file_update file(ino 16 size 0xc4d6 mtime 2021-03-08T09:54:03.075304+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.107 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x50000: txn(seq 81 len 0x30 crc 0x65b0d84) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x50000: op_file_update file(ino 16 size 0x10b5d mtime 2021-03-08T09:54:03.105708+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x51000~1000 from file(ino 1 size 0x51000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.108 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x64000: op_file_update file(ino 16 size 0x127f0 mtime 2021-03-08T09:54:11.122485+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x65000~1000 from file(ino 1 size 0x65000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0x9b000 len 0x1000 2021-03-08T10:01:42.108 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x14ad mtime 2021-03-08T09:54:14.368641+0000 allocated 10000 extents [1:0x490000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x72000~1000 from file(ino 1 size 0x72000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0x8e000 len 0x1000 2021-03-08T10:01:42.109 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs stat dir sharding not found , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (sequential) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578df2fb0 on file(ino 30 size 0x10 mtime 2021-03-08T09:54:16.033765+0000 allocated 10000 extents [1:0x430000~10000]) 2021-03-08T10:01:42.110 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.two_write_queues: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.manual_wal_flush: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.atomic_flush: 0 2021-03-08T10:01:42.110 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-08T10:01:42.111 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578de6380 on file(ino 31 size 0x17f mtime 2021-03-08T09:54:16.036433+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:17.404+0000 7f284002fdc0 10 bluefs _read h 0x563578de6380 0x0~8000 from file(ino 31 size 0x17f mtime 2021-03-08T09:54:16.036433+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:17.404+0000 7f284002fdc0 20 bluefs _read reaching (or past) eof, len clipped to 0x17f 2021-03-08T10:01:42.112 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0x1000: op_file_update file(ino 3 size 0x25 mtime 2021-03-08T09:53:37.917565+0000 allocated 10000 extents [1:0x410000~10000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x2000~1000 from file(ino 1 size 0x2000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read left 0xfe000 len 0x1000 2021-03-08T10:01:42.113 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0xf000: op_dir_link db/000007.sst to 13 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0xf000: op_file_update file(ino 13 size 0x4df mtime 2021-03-08T09:53:55.320697+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x10000~1000 from file(ino 1 size 0x10000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.113 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read left 0xe0000 len 0x1000 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _replay 0x20000: txn(seq 33 len 0x24 crc 0xe25a7e16) 2021-03-08T10:01:42.114 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _replay 0x34000: txn(seq 53 len 0x24 crc 0xe69616ee) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0x34000: op_file_update file(ino 16 size 0xa7f5 mtime 2021-03-08T09:54:03.069264+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.114 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _replay 0x48000: txn(seq 73 len 0x24 crc 0xeb96782) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x48000: op_file_update file(ino 16 size 0xe14b mtime 2021-03-08T09:54:03.083414+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x49000~1000 from file(ino 1 size 0x49000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.115 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x5c000: op_file_update file(ino 16 size 0x126ad mtime 2021-03-08T09:54:11.081077+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x5d000~1000 from file(ino 1 size 0x5d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _read left 0xa3000 len 0x1000 2021-03-08T10:01:42.116 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x6c000: op_dir_link db/000013.sst to 23 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x6c000: op_file_update file(ino 23 size 0x407 mtime 2021-03-08T09:54:14.367323+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x6d000~1000 from file(ino 1 size 0x6d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.116 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x78000: op_file_remove 24 , 2021-03-08T09:54:18.129 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pgid 2.3 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x79000~1000 from file(ino 1 size 0x79000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.117 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.create_if_missing: 0 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.paranoid_checks: 1 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.env: 0x563578de3120 2021-03-08T10:01:42.118 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (random) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578deeef0 on file(ino 30 size 0x10 mtime 2021-03-08T09:54:16.033765+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (sequential) 2021-03-08T10:01:42.118 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-08T10:01:42.119 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.671+0000 7f284002fdc0 10 bluefs _flush 0x563578c923f0 0x7d000~1000 to file(ino 1 size 0x7d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.671+0000 7f284002fdc0 10 bluefs _flush_range 0x563578c923f0 pos 0x7d000 0x7d000~1000 to file(ino 1 size 0x7d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.671+0000 7f284002fdc0 20 bluefs _flush_range file now file(ino 1 size 0x7e000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.120 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _flush_range h 0x563578c923f0 pos now 0x80000 , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 wait_for_aio 0x563578c923f0 2021-03-08T10:01:42.120 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _read left 0x2000 len 0x14ad , 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _read got 5293 , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 bluefs rename db/OPTIONS-000025.dbtmp -> db/OPTIONS-000026 2021-03-08T10:01:42.121 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread start , 2021-03-08T09:54:17.673+0000 7f27fcff9700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread sleep , 2021-03-08T09:54:17.673+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _deferred_replay start 2021-03-08T10:01:42.122 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f284002fdc0 15 bluestore(/var/lib/ceph/osd/ceph-0) omap_get_values 2.0_head oid #2:00000000::::head# , 2021-03-08T09:54:17.673+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.0_head 0x563578c35810) get_onode oid #2:00000000::::head# key 0x7F800000000000000200000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F , 2021-03-08T09:54:17.673+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.0_head 0x563578c35810) r 0 v.len 29 2021-03-08T10:01:42.122 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 wait_for_aio 0x563578dc8010 , 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluefs wait_for_aio 0x563578dc8010 done in 0.000021 2021-03-08T10:01:42.123 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.674+0000 7f281a7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_committed_kv txc 0x563578dc90f0 , 2021-03-08T09:54:17.674+0000 7f281a7fc700 10 bluestore(/var/lib/ceph/osd/ceph-0) _txc_state_proc txc 0x563578dc90f0 finishing , 2021-03-08T09:54:17.674+0000 7f281a7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish 0x563578dc90f0 onodes 0x563578dc9910 2021-03-08T10:01:42.124 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.674+0000 7f284002fdc0 20 bluefs _stop_alloc , 2021-03-08T09:54:17.674+0000 7f284002fdc0 1 bdev(0x563578d19b30 /var/lib/ceph/osd/ceph-0/block) close , 2021-03-08T09:54:17.939+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _close_fm 2021-03-08T10:01:42.124 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 10 bluefs _replay 0x7000: txn(seq 8 len 0x63 crc 0x113625d1) , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x7000: op_dir_link db/OPTIONS-000005 to 7 2021-03-08T10:01:42.125 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x0 mtime 2021-03-08T09:53:55.321586+0000 allocated 0 extents []) , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_dir_link db/OPTIONS-000010.dbtmp to 17 , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x14ad mtime 2021-03-08T09:53:55.321689+0000 allocated 10000 extents [1:0x470000~10000]) 2021-03-08T10:01:42.125 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _replay 0x27000: op_file_update file(ino 16 size 0x8c01 mtime 2021-03-08T09:54:02.070915+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x28000~1000 from file(ino 1 size 0x28000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xd8000 len 0x1000 2021-03-08T10:01:42.126 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x3c000~1000 from file(ino 1 size 0x3c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xc4000 len 0x1000 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.127 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xb0000 len 0x1000 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _replay 0x50000: txn(seq 81 len 0x30 crc 0x65b0d84) 2021-03-08T10:01:42.128 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _replay 0x64000: txn(seq 101 len 0x30 crc 0x7f93623a) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _replay 0x64000: op_file_update file(ino 16 size 0x127f0 mtime 2021-03-08T09:54:11.122485+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.128 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x0 mtime 2021-03-08T09:54:14.368528+0000 allocated 0 extents []) , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_dir_link db/OPTIONS-000016.dbtmp to 27 , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x14ad mtime 2021-03-08T09:54:14.368641+0000 allocated 10000 extents [1:0x490000~10000]) 2021-03-08T10:01:42.129 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x7e000: op_dir_link db/MANIFEST-000023 to 34 , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x7e000: op_file_update file(ino 34 size 0xd3 mtime 2021-03-08T09:54:17.672825+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x7f000~1000 from file(ino 1 size 0x7f000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.130 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 10 bluefs stat db/MANIFEST-000023 file(ino 34 size 0xd3 mtime 2021-03-08T09:54:17.672825+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: MANIFEST file: MANIFEST-000023 size: 211 Bytes , 2021-03-08T10:01:42.130 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x10 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 20 bluefs _read got 16 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54d160 0x10~2000 from file(ino 35 size 0x10 mtime 2021-03-08T09:54:17.672989+0000 allocated 10000 extents [1:0x4b0000~10000]) 2021-03-08T10:01:42.131 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-08T10:01:42.131 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 20 bluefs _read_random left 0xe39 0x41c7~8c , 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 10 bluefs _read_random h 0x7fccf8001d20 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) 2021-03-08T10:01:42.132 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 20 bluefs _replay 0x3000: op_file_update file(ino 5 size 0x10 mtime 2021-03-08T09:53:37.917928+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 20 bluefs _read left 0xfc000 len 0x1000 2021-03-08T10:01:42.133 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x12000~1000 from file(ino 1 size 0x12000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.133 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.134 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _replay 0x37000: txn(seq 56 len 0x24 crc 0x2df8dbf5) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.135 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x4b000: op_file_update file(ino 16 size 0xf629 mtime 2021-03-08T09:54:03.086479+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xb4000 len 0x1000 2021-03-08T10:01:42.135 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x60000~1000 from file(ino 1 size 0x60000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.136 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x6f000~1000 from file(ino 1 size 0x6f000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.137 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x7b000~1000 from file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.137 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a557940 0x10~2000 from file(ino 35 size 0x10 mtime 2021-03-08T09:54:17.672989+0000 allocated 10000 extents [1:0x4b0000~10000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 0 2021-03-08T10:01:42.138 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.max_background_jobs: 2 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-08T10:01:42.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.level: 32767 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-08T10:01:42.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x25 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~1000 of 1:0x410000~10000 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x25 2021-03-08T10:01:42.140 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluestore.BufferSpace(0x55fb4a55c5e8 in 0x55fb4a4a5a10) _clear , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection meta cnode(bits 0) , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #-1:3e70e826:::osdmap.12:0# 2021-03-08T10:01:42.140 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluefs _read_random got 3791 , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_on_open checking for stray omap data 2021-03-08T10:01:42.141 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.650+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x0~1000 from file(ino 1 size 0x1000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.650+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~100000 of 1:0x10000~400000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0x100000 len 0x1000 2021-03-08T10:01:42.142 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0xd000: txn(seq 14 len 0x60 crc 0xe012ceaf) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0xd000: op_file_update file(ino 12 size 0x0 mtime 2021-03-08T09:53:40.233346+0000 allocated 0 extents []) 2021-03-08T10:01:42.142 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x1c000: op_file_update file(ino 16 size 0x51ee mtime 2021-03-08T09:53:59.056942+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x1d000~1000 from file(ino 1 size 0x1d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xe3000 len 0x1000 2021-03-08T10:01:42.143 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x31000~1000 from file(ino 1 size 0x31000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xcf000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xbb000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x45000: txn(seq 70 len 0x24 crc 0x51520b6e) 2021-03-08T10:01:42.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x59000: txn(seq 90 len 0x30 crc 0x2ec16f6b) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x59000: op_file_update file(ino 16 size 0x1263b mtime 2021-03-08T09:54:11.078081+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_file_update file(ino 22 size 0x0 mtime 2021-03-08T09:54:12.770693+0000 allocated 0 extents []) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_dir_link db/OPTIONS-000013.dbtmp to 22 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_file_update file(ino 22 size 0x14ad mtime 2021-03-08T09:54:12.770777+0000 allocated 10000 extents [1:0x4e0000~10000]) 2021-03-08T10:01:42.146 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0x89000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x77000: txn(seq 120 len 0x4c crc 0xb16f6660) 2021-03-08T10:01:42.146 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 10 bluefs _replay 0x84000: txn(seq 133 len 0x23 crc 0xb910e119) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 20 bluefs _replay 0x84000: op_file_update file(ino 36 size 0x61f mtime 2021-03-08T09:54:17.675013+0000 allocated 10000 extents [1:0x4d0000~10000]) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x85000~1000 from file(ino 1 size 0x85000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.147 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.use_adaptive_mutex: 0 , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.rate_limiter: (nil) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-08T10:01:42.148 WARNING:scrape:Ignoring malparsed backtrace: whole_key_filtering: 1 , verify_compression: 0 , read_amp_bytes_per_bit: 0 2021-03-08T10:01:42.148 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 10 bluefs _read_random h 0x7fcd04005050 0x188~2b8 from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 20 bluefs _read_random left 0xe78 0x188~2b8 , 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 20 bluefs _read_random got 696 2021-03-08T10:01:42.149 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 10 bluefs _close_writer 0x55fb4a49c4d0 type 0 , 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 20 bluefs _stop_alloc , 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 1 bdev(0x55fb4a542a80 /var/lib/ceph/osd/ceph-0/block) close 2021-03-08T10:01:42.149 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 20 bluefs _replay 0xa000: op_dir_link db/000005.dbtmp to 10 , 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.150 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.151 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _replay 0x2c000: txn(seq 45 len 0x24 crc 0x5f1d3bb0) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.151 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x40000: op_file_update file(ino 16 size 0xd7e4 mtime 2021-03-08T09:54:03.076199+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0xbf000 len 0x1000 2021-03-08T10:01:42.152 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x55000~1000 from file(ino 1 size 0x55000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.153 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x68000~1000 from file(ino 1 size 0x68000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.153 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_file_remove 23 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_file_remove 18 2021-03-08T10:01:42.154 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_file_remove 31 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_file_remove 29 2021-03-08T10:01:42.155 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.info_log: 0x55fb4a47f510 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.max_file_opening_threads: 16 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.statistics: (nil) 2021-03-08T10:01:42.155 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x10 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~1000 of 1:0x4b0000~10000 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x10 2021-03-08T10:01:42.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-08T10:01:42.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _fsync 0x55fb4a4a43f0 file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _flush 0x55fb4a4a43f0 0x0~93d to file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _flush_range 0x55fb4a4a43f0 pos 0x0 0x0~93d to file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) 2021-03-08T10:01:42.157 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 10 HybridAllocator allocate want 0x10000 unit 0x10000 max_alloc_size 0x10000 hint 0x0 , 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 20 bluefs _flush_range dirty_seq = 136 (was clean) , 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 20 bluefs _flush_range file now file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) 2021-03-08T10:01:42.158 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range file now file(ino 1 size 0x8b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range in 1:0x10000~400000 x_off 0x8a000 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range h 0x55fb4a49c4d0 pos now 0x8b000 2021-03-08T10:01:42.158 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 10 bluefs _read_random h 0x7fccf8004c80 0x0~107b from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _read_random read random 0x0~107b of 1:0x4a0000~10000 2021-03-08T10:01:42.159 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd0e7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_finalize_thread sleep , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55fb4a56ce60 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all done 2021-03-08T10:01:42.160 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment do_bluefs = 1 , 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment initializing bluefs , 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block 0 2021-03-08T10:01:42.160 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_link db/CURRENT to 10 , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_unlink db/000005.dbtmp , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0xc000~1000 from file(ino 1 size 0xc000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.161 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.490+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x1a000~1000 from file(ino 1 size 0x1a000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _read left 0xe6000 len 0x1000 , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.162 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0xd2000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x2e000: txn(seq 47 len 0x24 crc 0xbc743dc4) 2021-03-08T10:01:42.162 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x42000: txn(seq 67 len 0x24 crc 0x851bc9b8) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x42000: op_file_update file(ino 16 size 0xd830 mtime 2021-03-08T09:54:03.077262+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.163 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x56000: txn(seq 87 len 0x30 crc 0xb9858be4) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x56000: op_file_update file(ino 16 size 0x125a3 mtime 2021-03-08T09:54:11.077692+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x57000~1000 from file(ino 1 size 0x57000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.163 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x69000~1000 from file(ino 1 size 0x69000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0x97000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.164 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0x8b000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x75000: txn(seq 118 len 0x5b crc 0xf271c68f) 2021-03-08T10:01:42.165 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_dir_link db/OPTIONS-000025.dbtmp to 37 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_file_update file(ino 37 size 0x14ad mtime 2021-03-08T09:54:17.673483+0000 allocated 10000 extents [1:0x4c0000~10000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x83000~1000 from file(ino 1 size 0x83000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.165 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 20 bluefs _read got 279 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 5 rocksdb: verify_sharding column families from rocksdb: [default] , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 10 bluefs open_for_read sharding/recreate_columns (sequential) 2021-03-08T10:01:42.166 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.stats_history_buffer_size: 1048576 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.max_open_files: -1 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.bytes_per_sync: 0 2021-03-08T10:01:42.167 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.target_file_size_multiplier: 1 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2021-03-08T10:01:42.167 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.493+0000 7ff6167fc700 10 bluefs _read_random h 0x7ff600001d20 0x11a~4a from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:20.493+0000 7ff6167fc700 20 bluefs _read_random left 0xee6 0x11a~4a , 2021-03-08T09:54:20.493+0000 7ff6167fc700 20 bluefs _read_random got 74 2021-03-08T10:01:42.168 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment initializing bluefs , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block 0 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 1 bdev(0x5646fca28a30 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2021-03-08T10:01:42.169 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_unlink db/000005.dbtmp , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0xc000~1000 from file(ino 1 size 0xc000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read left 0xf4000 len 0x1000 2021-03-08T10:01:42.169 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read left 0xe6000 len 0x1000 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs _replay 0x1a000: txn(seq 27 len 0x24 crc 0x91ca6867) 2021-03-08T10:01:42.170 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x2e000: txn(seq 47 len 0x24 crc 0xbc743dc4) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x2e000: op_file_update file(ino 16 size 0x9e52 mtime 2021-03-08T09:54:03.064379+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.170 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x42000: txn(seq 67 len 0x24 crc 0x851bc9b8) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x42000: op_file_update file(ino 16 size 0xd830 mtime 2021-03-08T09:54:03.077262+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x43000~1000 from file(ino 1 size 0x43000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.171 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x56000: op_file_update file(ino 16 size 0x125a3 mtime 2021-03-08T09:54:11.077692+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x57000~1000 from file(ino 1 size 0x57000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0xa9000 len 0x1000 2021-03-08T10:01:42.172 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0x97000 len 0x1000 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x69000: txn(seq 106 len 0x7c crc 0x67e69dad) 2021-03-08T10:01:42.172 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x75000: txn(seq 118 len 0x5b crc 0xf271c68f) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x75000: op_file_update file(ino 29 size 0x0 mtime 2021-03-08T09:54:16.033410+0000 allocated 0 extents []) 2021-03-08T10:01:42.173 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_file_update file(ino 37 size 0x14ad mtime 2021-03-08T09:54:17.673483+0000 allocated 10000 extents [1:0x4c0000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x83000~1000 from file(ino 1 size 0x83000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0x7d000 len 0x1000 2021-03-08T10:01:42.174 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 20 bluefs _read got 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 10 bluefs open_for_read db/MANIFEST-000026 (sequential) 2021-03-08T10:01:42.174 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.175 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-08T10:01:42.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 10 bluefs _read_random h 0x7ff600003ab0 0x58c~4a from file(ino 38 size 0x93d mtime 2021-03-08T09:54:19.917885+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 20 bluefs _read_random left 0xa74 0x58c~4a , 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 20 bluefs _read_random got 74 2021-03-08T10:01:42.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow 0x0~a59: 0x0~a59 Blob(0x5646fca2c190 blob([0xf000~1000] csum crc32c/0x1000) use_tracker(0x1000 0xa59) SharedBlob(0x5646fcac6970 sbid 0x0)) , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore.sharedblob(0x5646fcac6970) put 0x5646fcac6970 removing self from set 0x5646fcac6570 , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore.BufferSpace(0x5646fcac6988 in 0x5646fc98ba10) _clear 2021-03-08T10:01:42.177 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection 1.0_head cnode(bits 0) , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:00000000::::head# , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:d4465517:::INTEL_SSDPED1D480GA_PHMB7512009A480DGN:head# 2021-03-08T10:01:42.178 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.254+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:21.254+0000 7ff636d71dc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:21.254+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.178 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.179 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.180 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.180 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.181 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.182 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.182 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.183 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.184 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 20 bluefs _read got 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 10 bluefs open_for_read db/MANIFEST-000026 (sequential) , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 10 bluefs open_for_read h 0x5646fca12750 on file(ino 39 size 0x117 mtime 2021-03-08T09:54:19.918263+0000 allocated 10000 extents [1:0x430000~10000]) 2021-03-08T10:01:42.184 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-08T10:01:42.185 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.enabled: false 2021-03-08T10:01:42.185 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff6167fc700 20 bluefs _read_random got 85 , 2021-03-08T09:54:21.256+0000 7ff6167fc700 10 bluefs _read_random h 0x7ff604004aa0 0x188~2b8 from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:21.256+0000 7ff6167fc700 20 bluefs _read_random left 0xe78 0x188~2b8 2021-03-08T10:01:42.186 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.257+0000 7ff636d71dc0 1 bdev(0x5646fca28a30 /var/lib/ceph/osd/ceph-0/block) close , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) 2021-03-08T10:01:42.187 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xf5000 len 0x1000 , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.187 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xe7000 len 0x1000 2021-03-08T10:01:42.188 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xd3000 len 0x1000 , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.189 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x41000: txn(seq 66 len 0x24 crc 0x5306814b) 2021-03-08T10:01:42.189 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x55000: op_file_update file(ino 16 size 0x1257d mtime 2021-03-08T09:54:11.077559+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.190 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x68000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.191 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 13 2021-03-08T10:01:42.191 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read left 0x7e000 len 0x1000 2021-03-08T10:01:42.192 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs open_for_read h 0x5646fca0e110 on file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca0e110 0x0~2000 from file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x10 2021-03-08T10:01:42.192 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.atomic_flush: 0 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-08T10:01:42.193 WARNING:scrape:Ignoring malparsed backtrace: verify_compression: 0 , read_amp_bytes_per_bit: 0 , format_version: 2 2021-03-08T10:01:42.194 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.520+0000 7ff5f6ffd700 20 bluefs _read_random got 697 , 2021-03-08T09:54:21.520+0000 7ff616ffd700 20 bluefs _read left 0x1000 len 0x4ca , 2021-03-08T09:54:21.520+0000 7ff616ffd700 20 bluefs _read got 1226 2021-03-08T10:01:42.194 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 wait_for_aio 0x5646fc9824d0 , 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 bluefs wait_for_aio 0x5646fc9824d0 done in 0.000019 2021-03-08T10:01:42.195 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs _flush_range in 1:0x4b0000~10000 x_off 0x0 , 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs flush_buffer caching tail of 0x4ad and padding block with 0xb53 buffer.length() 0 , 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs _flush_range h 0x5646fc980510 pos now 0x14ad 2021-03-08T10:01:42.196 WARNING:scrape:Ignoring malparsed backtrace: Flush(GB): cumulative 0.000, interval 0.000 , AddFile(GB): cumulative 0.000, interval 0.000 , AddFile(Total Files): cumulative 0, interval 0 2021-03-08T10:01:42.196 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.522+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x5646fca2a830 , 2021-03-08T09:54:21.522+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x5646fca32650 , 2021-03-08T09:54:21.522+0000 7ff5fd7fa700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread sleep 2021-03-08T10:01:42.197 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.631+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:22.631+0000 7fb50e40adc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:22.631+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.197 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.198 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.199 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.199 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.200 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.201 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.201 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.202 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.203 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _replay 0x8f000: txn(seq 144 len 0x7c crc 0x97e17074) , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x8f000: op_file_update file(ino 45 size 0x0 mtime 2021-03-08T09:54:21.522082+0000 allocated 0 extents []) 2021-03-08T10:01:42.203 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.204 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.205 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs open_for_read db/000022.sst (random) , 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs open_for_read h 0x7fb4dc001d20 on file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs _read h 0x7fb4dc001d20 0x0~4ca from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) prefetch 2021-03-08T10:01:42.205 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.635+0000 7fb50e40adc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:22.635+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4d4001d20 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:22.635+0000 7fb50e40adc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 2021-03-08T10:01:42.206 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.895+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.206 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) 2021-03-08T10:01:42.207 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.208 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xc8000 len 0x1000 2021-03-08T10:01:42.208 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.209 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) 2021-03-08T10:01:42.210 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) 2021-03-08T10:01:42.210 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) 2021-03-08T10:01:42.211 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x89000: txn(seq 138 len 0x7c crc 0x140649b0) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_file_update file(ino 41 size 0x0 mtime 2021-03-08T09:54:19.918678+0000 allocated 0 extents []) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_dir_link db.wal/000027.log to 41 2021-03-08T10:01:42.211 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 0 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs open_for_read db/MANIFEST-000029 (sequential) 2021-03-08T10:01:42.212 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.213 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-08T10:01:42.213 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb4f67fc700 20 bluefs _read_random left 0xa06 0x5fa~2b9 , 2021-03-08T09:54:22.897+0000 7fb4f67fc700 20 bluefs _read_random got 697 , 2021-03-08T09:54:22.897+0000 7fb4ca7fc700 10 bluefs _read_random h 0x7fb4e0003ac0 0x11a~4a from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) 2021-03-08T10:01:42.214 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow 0x0~a59: 0x0~a59 Blob(0x55eec084d190 blob([0xf000~1000] csum crc32c/0x1000) use_tracker(0x1000 0xa59) SharedBlob(0x55eec0863c00 sbid 0x0)) , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore.sharedblob(0x55eec0863c00) put 0x55eec0863c00 removing self from set 0x55eec085e460 , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore.BufferSpace(0x55eec0863c18 in 0x55eec07aca10) _clear 2021-03-08T10:01:42.215 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection 1.0_head cnode(bits 0) , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:00000000::::head# , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:d4465517:::INTEL_SSDPED1D480GA_PHMB7512009A480DGN:head# 2021-03-08T10:01:42.215 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.399+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:23.399+0000 7fb50e40adc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:23.399+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.216 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.217 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.217 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.218 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.219 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.219 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.220 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.221 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.221 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 10 bluefs _replay 0x8f000: txn(seq 144 len 0x7c crc 0x97e17074) , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 20 bluefs _replay 0x8f000: op_file_update file(ino 45 size 0x0 mtime 2021-03-08T09:54:21.522082+0000 allocated 0 extents []) 2021-03-08T10:01:42.222 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.222 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.223 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs open_for_read db/000022.sst (random) , 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs open_for_read h 0x7fb4d4004c60 on file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs _read h 0x7fb4d4004c60 0x0~4ca from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) prefetch 2021-03-08T10:01:42.224 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.402+0000 7fb50e40adc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:23.402+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4e00039c0 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:23.402+0000 7fb50e40adc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 2021-03-08T10:01:42.224 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.663+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.225 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) 2021-03-08T10:01:42.226 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.226 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xc8000 len 0x1000 2021-03-08T10:01:42.227 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.228 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) 2021-03-08T10:01:42.228 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) 2021-03-08T10:01:42.229 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) 2021-03-08T10:01:42.229 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x89000: txn(seq 138 len 0x7c crc 0x140649b0) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_file_update file(ino 41 size 0x0 mtime 2021-03-08T09:54:19.918678+0000 allocated 0 extents []) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_dir_link db.wal/000027.log to 41 2021-03-08T10:01:42.230 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 0 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs open_for_read db/MANIFEST-000029 (sequential) 2021-03-08T10:01:42.231 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.231 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.max_write_buffer_number: 4 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.compression: NoCompression , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-08T10:01:42.232 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb4f5ffb700 20 bluefs _read_random got 53 , 2021-03-08T09:54:23.665+0000 7fb4f67fc700 10 bluefs _read_random h 0x7fb4d4005cd0 0x908~35 from file(ino 38 size 0x93d mtime 2021-03-08T09:54:19.917885+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:23.665+0000 7fb4f67fc700 20 bluefs _read_random left 0x6f8 0x908~35 2021-03-08T10:01:42.233 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.666+0000 7fb50e40adc0 10 bluefs open_for_write db/000032.dbtmp , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs open_for_write mapping db/000032.dbtmp vsel_hint 0x1 , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 10 bluefs open_for_write h 0x55eec0879010 on file(ino 48 size 0x0 mtime 2021-03-08T09:54:23.667013+0000 allocated 0 extents []) 2021-03-08T10:01:42.233 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _fsync file metadata was dirty (151) on file(ino 50 size 0x14ad mtime 2021-03-08T09:54:23.667558+0000 allocated 10000 extents [1:0x490000~10000]), flushing log , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _flush_and_sync_log 1 dirty_files , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _flush_and_sync_log op_file_update file(ino 50 size 0x14ad mtime 2021-03-08T09:54:23.667558+0000 allocated 10000 extents [1:0x490000~10000]) 2021-03-08T10:01:42.234 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluefs _read_random got 282 , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4e0003ba0 0x41c7~8c from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluefs _read_random left 0xe39 0x41c7~8c 2021-03-08T10:01:42.235 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55eec085de00 , 2021-03-08T09:54:23.667+0000 7fb4f67fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_finalize_thread sleep , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55eec085e1b0 2021-03-08T10:01:42.235 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:34.490 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] , 2021-03-08T09:54:39.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2021-03-08T10:01:42.236 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:56:47.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2021-03-08T09:56:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-08T10:01:42.236 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2021-03-08T09:58:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2021-03-08T09:58:56.167 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-03-08T10:01:42.238 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/ceph_manager.py", line 173, in wrapper , return func(self) , File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/ceph_manager.py", line 1261, in _do_thrash 2021-03-08T10:01:42.239 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:00:48.076 INFO:teuthology.orchestra.run.smithi046.stdout:{ , 2021-03-08T10:00:48.077 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "" , 2021-03-08T10:00:48.077 INFO:teuthology.orchestra.run.smithi046.stdout:} 2021-03-08T10:01:42.239 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:01:10.739 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com , 2021-03-08T10:01:10.740 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2021-03-08T10:01:10.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2021-03-08T10:01:42.240 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:01:12.932 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-03-08T10:01:12.933 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com , 2021-03-08T10:01:12.933 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-03-08T10:01:42.283 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1646-g1cbcfeff (1cbcfeffb31882cc49baf3f06acedb4bb72a3ebc) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7fce5077b638] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x560fff894552] 2021-03-08T10:01:42.285 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x8000: txn(seq 9 len 0x56 crc 0x167e249) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x8000: op_file_update file(ino 8 size 0x0 mtime 2021-03-08T09:53:40.232055+0000 allocated 0 extents []) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x8000: op_dir_link db/000004.sst to 8 2021-03-08T10:01:42.285 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_dir_unlink db/OPTIONS-000010.dbtmp , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_dir_unlink db/OPTIONS-000005 , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x15000: op_file_remove 7 2021-03-08T10:01:42.286 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x29000: txn(seq 42 len 0x24 crc 0x50f08ff2) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x29000: op_file_update file(ino 16 size 0x8c4d mtime 2021-03-08T09:54:02.071826+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.287 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _replay 0x3d000: txn(seq 62 len 0x24 crc 0x683cf166) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 20 bluefs _replay 0x3d000: op_file_update file(ino 16 size 0xc9dc mtime 2021-03-08T09:54:03.075552+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:15.766+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x3e000~1000 from file(ino 1 size 0x3e000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.287 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x51000: op_file_update file(ino 16 size 0x10f69 mtime 2021-03-08T09:54:03.105950+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x52000~1000 from file(ino 1 size 0x52000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _read left 0xae000 len 0x1000 2021-03-08T10:01:42.288 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x65000: op_dir_link db/000010.sst to 18 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x65000: op_file_update file(ino 18 size 0x5557 mtime 2021-03-08T09:54:12.769670+0000 allocated 10000 extents [1:0x4b0000~10000]) , 2021-03-08T09:54:15.767+0000 7f27480dadc0 10 bluefs _read h 0x560fa768a620 0x66000~1000 from file(ino 1 size 0x66000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.289 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_dir_unlink db/OPTIONS-000016.dbtmp , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_dir_unlink db/OPTIONS-000011 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 20 bluefs _replay 0x72000: op_file_remove 17 2021-03-08T10:01:42.289 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:15.767+0000 7f27480dadc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.290 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.291 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs open_for_read db/000018.sst (random) , 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs open_for_read h 0x7f2718004f20 on file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:15.768+0000 7f2723fff700 10 bluefs _read h 0x7f2718004f20 0x35b3~1000 from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) prefetch 2021-03-08T10:01:42.291 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment kv_backend = rocksdb , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.292 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.029+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.029+0000 7f27480dadc0 10 bluefs _replay 0xb000: txn(seq 12 len 0x4c crc 0x3752ac14) , 2021-03-08T09:54:16.029+0000 7f27480dadc0 20 bluefs _replay 0xb000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.293 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read left 0xe7000 len 0x1000 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x19000: txn(seq 26 len 0x24 crc 0xac68b3d8) 2021-03-08T10:01:42.294 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x2d000: txn(seq 46 len 0x24 crc 0x15e3ab73) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x2d000: op_file_update file(ino 16 size 0x8ce5 mtime 2021-03-08T09:54:02.078862+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.295 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _replay 0x41000: txn(seq 66 len 0x24 crc 0x5306814b) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x41000: op_file_update file(ino 16 size 0xd80a mtime 2021-03-08T09:54:03.076341+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _read h 0x560fa7686bf0 0x42000~1000 from file(ino 1 size 0x42000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.296 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x55000: op_file_update file(ino 16 size 0x1257d mtime 2021-03-08T09:54:11.077559+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 10 bluefs _read h 0x560fa7686bf0 0x56000~1000 from file(ino 1 size 0x56000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _read left 0xaa000 len 0x1000 2021-03-08T10:01:42.296 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_dir_unlink db/CURRENT , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_file_remove 15 , 2021-03-08T09:54:16.030+0000 7f27480dadc0 20 bluefs _replay 0x68000: op_dir_link db/CURRENT to 20 2021-03-08T10:01:42.297 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_file_remove 13 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_dir_unlink db/000004.sst , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _replay 0x74000: op_file_remove 8 2021-03-08T10:01:42.298 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.manifest_preallocation_size: 4194304 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.is_fd_close_on_exec: 1 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 4 rocksdb: Options.advise_random_on_open: 1 2021-03-08T10:01:42.299 WARNING:scrape:Ignoring malparsed backtrace: pin_top_level_index_and_filter: 1 , index_type: 0 , data_block_index_type: 0 2021-03-08T10:01:42.299 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read reaching (or past) eof, len clipped to 0x1bf , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read fetching 0x0~1000 of 1:0x450000~10000 , 2021-03-08T09:54:16.031+0000 7f27480dadc0 20 bluefs _read left 0x1000 len 0x1bf 2021-03-08T10:01:42.300 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.032+0000 7f27480dadc0 10 wait_for_aio 0x560fa75d9810 , 2021-03-08T09:54:16.032+0000 7f27480dadc0 10 bluefs wait_for_aio 0x560fa75d9810 done in 0.000018 , 2021-03-08T09:54:16.032+0000 7f27480dadc0 20 bluefs flush_bdev 2021-03-08T10:01:42.301 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.033+0000 7f27480dadc0 20 bluefs _flush_and_sync_log cleaned file file(ino 32 size 0x14ad mtime 2021-03-08T09:54:16.034455+0000 allocated 10000 extents [1:0x450000~10000]) , 2021-03-08T09:54:16.033+0000 7f27480dadc0 10 bluefs _should_compact_log current 0x7a000 expected 3000 ratio 40.6667 , 2021-03-08T09:54:16.033+0000 7f27480dadc0 10 bluefs _flush 0x560fa75d5510 ignoring, length 0 < min_flush_size 524288 2021-03-08T10:01:42.302 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.034+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach 2.7_head fresh osr 0x560fa76a8090 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta 0x560fa768a3a0 cnode(bits 0) , 2021-03-08T09:54:16.034+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach meta fresh osr 0x560fa79e82d0 2021-03-08T10:01:42.303 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.3_head 0x560fa75c9c60) r 0 v.len 29 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 bluestore.OnodeSpace(0x560fa75c9da0 in 0x560fa7679380) add #2:c0000000::::head# 0x560fa7698250 , 2021-03-08T09:54:16.034+0000 7f27480dadc0 20 _add 0x560fa7679380 #2:c0000000::::head# added, num=6 2021-03-08T10:01:42.304 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27167fc700 20 bluefs _flush_range h 0x560fa75d9810 pos now 0x67 , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 wait_for_aio 0x560fa75d9810 2021-03-08T10:01:42.305 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finalize_kv txc 0x560fa79e8a20 allocated 0x[] released 0x[] , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _flush 0x560fa7680010 0x7b000~1000 to file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.035+0000 7f27167fc700 10 bluefs _flush_range 0x560fa7680010 pos 0x7b000 0x7b000~1000 to file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.306 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.035+0000 7f27480dadc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all , 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all osr_set 0x560fa75e8270,0x560fa7665f10,0x560fa76662c0,0x560fa7692ba0,0x560fa7692d80,0x560fa76945a0,0x560fa76a8090,0x560fa79e82d0 , 2021-03-08T09:54:16.035+0000 7f27480dadc0 20 bluestore(/var/lib/ceph/osd/ceph-0) deferred_try_submit 0 osrs, 0 txcs 2021-03-08T10:01:42.307 WARNING:scrape:Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/rados.py", line 277, in task 2021-03-08T10:01:42.308 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.635+0000 7f282b7fe700 10 -- >> v2:172.21.15.46:3300/0 conn(0x563578db1400 msgr2=0x563578db37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 26 remaining bytes 0 , 2021-03-08T09:54:16.635+0000 7f282bfff700 20 --1- >> v1:172.21.15.46:6789/0 conn(0x563578dacfb0 0x563578dad3f0 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write r=0 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 -- >> v2:172.21.15.46:3300/0 conn(0x563578db1400 msgr2=0x563578db37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=10 2021-03-08T10:01:42.309 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 10 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=6 , 2021-03-08T09:54:16.635+0000 7f282b7fe700 20 --2- 172.21.15.46:0/3256492216 >> v2:172.21.15.46:3300/0 conn(0x563578db1400 0x563578db37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done payload.length()=402 2021-03-08T10:01:42.309 WARNING:scrape:Ignoring malparsed backtrace: epoch 1 , fsid 4ab3c265-2973-4ab7-aa95-f86b36edad73 , last_changed 2021-03-08T09:53:36.414452+0000 2021-03-08T10:01:42.310 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.636+0000 7f284002fdc0 5 --2- 172.21.15.46:0/3256492216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563578db1400 0x563578db37f0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state , 2021-03-08T09:54:16.636+0000 7f284002fdc0 20 Event(0x563578d7c920 nevent=5000 time_id=3).wakeup , 2021-03-08T09:54:16.636+0000 7f284002fdc0 10 --2- 172.21.15.46:0/3256492216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563578db1400 0x563578db37f0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started 2021-03-08T10:01:42.311 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _replay 0x4000: txn(seq 5 len 0x31 crc 0x6c40b5c0) 2021-03-08T10:01:42.313 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _replay 0x12000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.313 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _read h 0x563578db17e0 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xdc000 len 0x1000 2021-03-08T10:01:42.314 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.638+0000 7f284002fdc0 10 bluefs _read h 0x563578db17e0 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read left 0xc8000 len 0x1000 , 2021-03-08T09:54:16.638+0000 7f284002fdc0 20 bluefs _read got 4096 2021-03-08T10:01:42.315 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x4c000: txn(seq 77 len 0x24 crc 0xae9c2ac6) 2021-03-08T10:01:42.315 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x60000: op_file_update file(ino 16 size 0x12745 mtime 2021-03-08T09:54:11.086495+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.316 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x6f000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.317 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.639+0000 7f284002fdc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) , 2021-03-08T09:54:16.639+0000 7f284002fdc0 20 bluefs _replay 0x7b000: op_file_update file(ino 31 size 0x14d mtime 2021-03-08T09:54:16.036261+0000 allocated 10000 extents [1:0x460000~10000]) 2021-03-08T10:01:42.317 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.advise_random_on_open: 1 , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.db_write_buffer_size: 0 , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: Options.write_buffer_manager: 0x563578de22f0 2021-03-08T10:01:42.318 WARNING:scrape:Ignoring malparsed backtrace: block_restart_interval: 16 , index_block_restart_interval: 1 , metadata_block_size: 4096 2021-03-08T10:01:42.319 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: [version_set.cc:4558] Recovered from manifest file:db/MANIFEST-000020 succeeded,manifest_file_number is 20, next_file_number is 22, last_sequence is 222, log_number is 16,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 , , 2021-03-08T09:54:16.640+0000 7f284002fdc0 4 rocksdb: [version_set.cc:4574] Column family [default] (ID 0), log number is 16 2021-03-08T10:01:42.319 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 10 bluefs _open_super log_fnode file(ino 1 size 0x1000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _init_alloc , 2021-03-08T09:54:16.902+0000 7f284002fdc0 1 bluefs _init_alloc shared, id 1, capacity 0x1680000000, block size 0x10000 2021-03-08T10:01:42.320 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_file_remove 6 , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_dir_unlink db/MANIFEST-000001 , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0xc000: op_file_remove 4 2021-03-08T10:01:42.321 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _replay 0x1b000: op_file_update file(ino 16 size 0x51c8 mtime 2021-03-08T09:53:59.056543+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x1c000~1000 from file(ino 1 size 0x1c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.902+0000 7f284002fdc0 20 bluefs _read left 0xe4000 len 0x1000 2021-03-08T10:01:42.322 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x30000~1000 from file(ino 1 size 0x30000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0xd0000 len 0x1000 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 2021-03-08T10:01:42.323 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0xbc000 len 0x1000 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x44000: txn(seq 69 len 0x24 crc 0xc5278ae2) 2021-03-08T10:01:42.324 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x58000: txn(seq 89 len 0x30 crc 0x6c84aec5) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x58000: op_file_update file(ino 16 size 0x12615 mtime 2021-03-08T09:54:11.077952+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.325 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x69000: op_file_remove 14 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _read h 0x563578defc60 0x6a000~1000 from file(ino 1 size 0x6a000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _read left 0x96000 len 0x1000 2021-03-08T10:01:42.326 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs _replay 0x76000: txn(seq 119 len 0x57 crc 0x50b5bec4) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x76000: op_file_update file(ino 30 size 0x0 mtime 2021-03-08T09:54:16.033729+0000 allocated 0 extents []) , 2021-03-08T09:54:16.903+0000 7f284002fdc0 20 bluefs _replay 0x76000: op_dir_link db/000020.dbtmp to 30 2021-03-08T10:01:42.326 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: DB SUMMARY , , 2021-03-08T09:54:16.903+0000 7f284002fdc0 10 bluefs readdir db 2021-03-08T10:01:42.327 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: kSnappyCompression supported: 1 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Fast CRC32 supported: Supported on x86 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: [db_impl/db_impl_readonly.cc:22] Opening the db in read only mode 2021-03-08T10:01:42.328 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 , 2021-03-08T09:54:16.903+0000 7f284002fdc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-08T10:01:42.329 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.904+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta 0x563578e013b0 cnode(bits 0) , 2021-03-08T09:54:16.904+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_attach meta fresh osr 0x563578e01580 , 2021-03-08T09:54:16.904+0000 7f284002fdc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_on_open <<>> check (regular) start 2021-03-08T10:01:42.330 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.908+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #-1:8bb276bf:::osdmap.7:0# , 2021-03-08T09:54:16.908+0000 7f284002fdc0 20 bluestore.blob(0x563578de2740) get_ref 0x0~538 Blob(0x563578de2740 blob([0x9000~1000] csum crc32c/0x1000) use_tracker(0x0 0x0) SharedBlob(0x563578de77b0 sbid 0x0)) , 2021-03-08T09:54:16.908+0000 7f284002fdc0 20 bluestore.blob(0x563578de2740) get_ref init 0x1000, 1000 2021-03-08T10:01:42.331 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578c8a110 in 0x563578d16140) clear 0 , 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578e00d90 in 0x563578d16140) clear 0 , 2021-03-08T09:54:16.982+0000 7f284002fdc0 10 bluestore.OnodeSpace(0x563578c7cda0 in 0x563578d16140) clear 0 2021-03-08T10:01:42.331 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_dir_link db/OPTIONS-000005 to 7 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_dir_unlink db/OPTIONS-000004.dbtmp , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x7000: op_file_update file(ino 6 size 0x133 mtime 2021-03-08T09:53:37.920382+0000 allocated 10000 extents [1:0x450000~10000]) 2021-03-08T10:01:42.332 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x14ad mtime 2021-03-08T09:53:55.321689+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x15000~1000 from file(ino 1 size 0x15000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0xeb000 len 0x1000 2021-03-08T10:01:42.333 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0xd8000 len 0x1000 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x28000: txn(seq 41 len 0x24 crc 0x1b12e88d) 2021-03-08T10:01:42.334 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x3c000: txn(seq 61 len 0x24 crc 0xf599ecfa) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x3c000: op_file_update file(ino 16 size 0xc4d6 mtime 2021-03-08T09:54:03.075304+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.335 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _replay 0x50000: txn(seq 81 len 0x30 crc 0x65b0d84) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x50000: op_file_update file(ino 16 size 0x10b5d mtime 2021-03-08T09:54:03.105708+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x51000~1000 from file(ino 1 size 0x51000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x64000: op_file_update file(ino 16 size 0x127f0 mtime 2021-03-08T09:54:11.122485+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x65000~1000 from file(ino 1 size 0x65000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0x9b000 len 0x1000 2021-03-08T10:01:42.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x14ad mtime 2021-03-08T09:54:14.368641+0000 allocated 10000 extents [1:0x490000~10000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs _read h 0x563578df2770 0x72000~1000 from file(ino 1 size 0x72000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs _read left 0x8e000 len 0x1000 2021-03-08T10:01:42.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.403+0000 7f284002fdc0 20 bluefs stat dir sharding not found , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (sequential) , 2021-03-08T09:54:17.403+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578df2fb0 on file(ino 30 size 0x10 mtime 2021-03-08T09:54:16.033765+0000 allocated 10000 extents [1:0x430000~10000]) 2021-03-08T10:01:42.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.two_write_queues: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.manual_wal_flush: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.atomic_flush: 0 2021-03-08T10:01:42.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 , 2021-03-08T09:54:17.404+0000 7f284002fdc0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-08T10:01:42.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.404+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578de6380 on file(ino 31 size 0x17f mtime 2021-03-08T09:54:16.036433+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:17.404+0000 7f284002fdc0 10 bluefs _read h 0x563578de6380 0x0~8000 from file(ino 31 size 0x17f mtime 2021-03-08T09:54:16.036433+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:17.404+0000 7f284002fdc0 20 bluefs _read reaching (or past) eof, len clipped to 0x17f 2021-03-08T10:01:42.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0x1000: op_file_update file(ino 3 size 0x25 mtime 2021-03-08T09:53:37.917565+0000 allocated 10000 extents [1:0x410000~10000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x2000~1000 from file(ino 1 size 0x2000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read left 0xfe000 len 0x1000 2021-03-08T10:01:42.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0xf000: op_dir_link db/000007.sst to 13 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0xf000: op_file_update file(ino 13 size 0x4df mtime 2021-03-08T09:53:55.320697+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x10000~1000 from file(ino 1 size 0x10000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read left 0xe0000 len 0x1000 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _replay 0x20000: txn(seq 33 len 0x24 crc 0xe25a7e16) 2021-03-08T10:01:42.343 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _read got 4096 , 2021-03-08T09:54:17.669+0000 7f284002fdc0 10 bluefs _replay 0x34000: txn(seq 53 len 0x24 crc 0xe69616ee) , 2021-03-08T09:54:17.669+0000 7f284002fdc0 20 bluefs _replay 0x34000: op_file_update file(ino 16 size 0xa7f5 mtime 2021-03-08T09:54:03.069264+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.345 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _replay 0x48000: txn(seq 73 len 0x24 crc 0xeb96782) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x48000: op_file_update file(ino 16 size 0xe14b mtime 2021-03-08T09:54:03.083414+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x49000~1000 from file(ino 1 size 0x49000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.345 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x5c000: op_file_update file(ino 16 size 0x126ad mtime 2021-03-08T09:54:11.081077+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x5d000~1000 from file(ino 1 size 0x5d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _read left 0xa3000 len 0x1000 2021-03-08T10:01:42.346 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x6c000: op_dir_link db/000013.sst to 23 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x6c000: op_file_update file(ino 23 size 0x407 mtime 2021-03-08T09:54:14.367323+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x6d000~1000 from file(ino 1 size 0x6d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 20 bluefs _replay 0x78000: op_file_remove 24 , 2021-03-08T09:54:18.129 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pgid 2.3 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs _read h 0x563578df4900 0x79000~1000 from file(ino 1 size 0x79000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.348 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.create_if_missing: 0 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.paranoid_checks: 1 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.env: 0x563578de3120 2021-03-08T10:01:42.348 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (random) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read h 0x563578deeef0 on file(ino 30 size 0x10 mtime 2021-03-08T09:54:16.033765+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:17.670+0000 7f284002fdc0 10 bluefs open_for_read db/CURRENT (sequential) 2021-03-08T10:01:42.349 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 , 2021-03-08T09:54:17.670+0000 7f284002fdc0 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-08T10:01:42.350 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.671+0000 7f284002fdc0 10 bluefs _flush 0x563578c923f0 0x7d000~1000 to file(ino 1 size 0x7d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.671+0000 7f284002fdc0 10 bluefs _flush_range 0x563578c923f0 pos 0x7d000 0x7d000~1000 to file(ino 1 size 0x7d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:17.671+0000 7f284002fdc0 20 bluefs _flush_range file now file(ino 1 size 0x7e000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.350 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _flush_range h 0x563578c923f0 pos now 0x80000 , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 wait_for_aio 0x563578c923f0 2021-03-08T10:01:42.351 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _read left 0x2000 len 0x14ad , 2021-03-08T09:54:17.672+0000 7f284002fdc0 20 bluefs _read got 5293 , 2021-03-08T09:54:17.672+0000 7f284002fdc0 10 bluefs rename db/OPTIONS-000025.dbtmp -> db/OPTIONS-000026 2021-03-08T10:01:42.352 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread start , 2021-03-08T09:54:17.673+0000 7f27fcff9700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread sleep , 2021-03-08T09:54:17.673+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _deferred_replay start 2021-03-08T10:01:42.352 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f284002fdc0 15 bluestore(/var/lib/ceph/osd/ceph-0) omap_get_values 2.0_head oid #2:00000000::::head# , 2021-03-08T09:54:17.673+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.0_head 0x563578c35810) get_onode oid #2:00000000::::head# key 0x7F800000000000000200000000'!!='0xFFFFFFFFFFFFFFFEFFFFFFFFFFFFFFFF6F , 2021-03-08T09:54:17.673+0000 7f284002fdc0 20 bluestore(/var/lib/ceph/osd/ceph-0).collection(2.0_head 0x563578c35810) r 0 v.len 29 2021-03-08T10:01:42.353 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 wait_for_aio 0x563578dc8010 , 2021-03-08T09:54:17.673+0000 7f27fcff9700 10 bluefs wait_for_aio 0x563578dc8010 done in 0.000021 2021-03-08T10:01:42.354 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.674+0000 7f281a7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_committed_kv txc 0x563578dc90f0 , 2021-03-08T09:54:17.674+0000 7f281a7fc700 10 bluestore(/var/lib/ceph/osd/ceph-0) _txc_state_proc txc 0x563578dc90f0 finishing , 2021-03-08T09:54:17.674+0000 7f281a7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _txc_finish 0x563578dc90f0 onodes 0x563578dc9910 2021-03-08T10:01:42.355 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:17.674+0000 7f284002fdc0 20 bluefs _stop_alloc , 2021-03-08T09:54:17.674+0000 7f284002fdc0 1 bdev(0x563578d19b30 /var/lib/ceph/osd/ceph-0/block) close , 2021-03-08T09:54:17.939+0000 7f284002fdc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _close_fm 2021-03-08T10:01:42.355 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 10 bluefs _replay 0x7000: txn(seq 8 len 0x63 crc 0x113625d1) , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x7000: op_dir_link db/OPTIONS-000005 to 7 2021-03-08T10:01:42.356 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x0 mtime 2021-03-08T09:53:55.321586+0000 allocated 0 extents []) , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_dir_link db/OPTIONS-000010.dbtmp to 17 , 2021-03-08T09:54:18.884+0000 7fcd2e001dc0 20 bluefs _replay 0x14000: op_file_update file(ino 17 size 0x14ad mtime 2021-03-08T09:53:55.321689+0000 allocated 10000 extents [1:0x470000~10000]) 2021-03-08T10:01:42.357 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _replay 0x27000: op_file_update file(ino 16 size 0x8c01 mtime 2021-03-08T09:54:02.070915+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x28000~1000 from file(ino 1 size 0x28000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xd8000 len 0x1000 2021-03-08T10:01:42.357 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x3c000~1000 from file(ino 1 size 0x3c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xc4000 len 0x1000 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.358 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read left 0xb0000 len 0x1000 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _replay 0x50000: txn(seq 81 len 0x30 crc 0x65b0d84) 2021-03-08T10:01:42.359 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 10 bluefs _replay 0x64000: txn(seq 101 len 0x30 crc 0x7f93623a) , 2021-03-08T09:54:18.885+0000 7fcd2e001dc0 20 bluefs _replay 0x64000: op_file_update file(ino 16 size 0x127f0 mtime 2021-03-08T09:54:11.122485+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.359 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x0 mtime 2021-03-08T09:54:14.368528+0000 allocated 0 extents []) , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_dir_link db/OPTIONS-000016.dbtmp to 27 , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x71000: op_file_update file(ino 27 size 0x14ad mtime 2021-03-08T09:54:14.368641+0000 allocated 10000 extents [1:0x490000~10000]) 2021-03-08T10:01:42.360 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x7e000: op_dir_link db/MANIFEST-000023 to 34 , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 20 bluefs _replay 0x7e000: op_file_update file(ino 34 size 0xd3 mtime 2021-03-08T09:54:17.672825+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.886+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a544960 0x7f000~1000 from file(ino 1 size 0x7f000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.361 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 10 bluefs stat db/MANIFEST-000023 file(ino 34 size 0xd3 mtime 2021-03-08T09:54:17.672825+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: MANIFEST file: MANIFEST-000023 size: 211 Bytes , 2021-03-08T10:01:42.361 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x10 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 20 bluefs _read got 16 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54d160 0x10~2000 from file(ino 35 size 0x10 mtime 2021-03-08T09:54:17.672989+0000 allocated 10000 extents [1:0x4b0000~10000]) 2021-03-08T10:01:42.362 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 , 2021-03-08T09:54:18.887+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-08T10:01:42.363 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 20 bluefs _read_random left 0xe39 0x41c7~8c , 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:18.888+0000 7fcd2e001dc0 10 bluefs _read_random h 0x7fccf8001d20 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) 2021-03-08T10:01:42.363 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 20 bluefs _replay 0x3000: op_file_update file(ino 5 size 0x10 mtime 2021-03-08T09:53:37.917928+0000 allocated 10000 extents [1:0x430000~10000]) , 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.149+0000 7fcd2e001dc0 20 bluefs _read left 0xfc000 len 0x1000 2021-03-08T10:01:42.364 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x12000~1000 from file(ino 1 size 0x12000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.365 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.366 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _replay 0x37000: txn(seq 56 len 0x24 crc 0x2df8dbf5) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.366 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _replay 0x4b000: op_file_update file(ino 16 size 0xf629 mtime 2021-03-08T09:54:03.086479+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xb4000 len 0x1000 2021-03-08T10:01:42.367 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x60000~1000 from file(ino 1 size 0x60000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:19.150+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.368 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x6f000~1000 from file(ino 1 size 0x6f000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.369 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54cf20 0x7b000~1000 from file(ino 1 size 0x7b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.370 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a557940 0x10~2000 from file(ino 35 size 0x10 mtime 2021-03-08T09:54:17.672989+0000 allocated 10000 extents [1:0x4b0000~10000]) , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:19.151+0000 7fcd2e001dc0 20 bluefs _read got 0 2021-03-08T10:01:42.370 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.max_background_jobs: 2 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-08T10:01:42.377 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.level: 32767 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-08T10:01:42.379 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x25 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~1000 of 1:0x410000~10000 , 2021-03-08T09:54:19.152+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x25 2021-03-08T10:01:42.380 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluestore.BufferSpace(0x55fb4a55c5e8 in 0x55fb4a4a5a10) _clear , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection meta cnode(bits 0) , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #-1:3e70e826:::osdmap.12:0# 2021-03-08T10:01:42.381 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 20 bluefs _read_random got 3791 , 2021-03-08T09:54:19.157+0000 7fcd2e001dc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_on_open checking for stray omap data 2021-03-08T10:01:42.382 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.650+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x0~1000 from file(ino 1 size 0x1000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.650+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~100000 of 1:0x10000~400000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0x100000 len 0x1000 2021-03-08T10:01:42.383 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0xd000: txn(seq 14 len 0x60 crc 0xe012ceaf) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0xd000: op_file_update file(ino 12 size 0x0 mtime 2021-03-08T09:53:40.233346+0000 allocated 0 extents []) 2021-03-08T10:01:42.384 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x1c000: op_file_update file(ino 16 size 0x51ee mtime 2021-03-08T09:53:59.056942+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x1d000~1000 from file(ino 1 size 0x1d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xe3000 len 0x1000 2021-03-08T10:01:42.385 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x31000~1000 from file(ino 1 size 0x31000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xcf000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.386 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0xbb000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x45000: txn(seq 70 len 0x24 crc 0x51520b6e) 2021-03-08T10:01:42.387 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x59000: txn(seq 90 len 0x30 crc 0x2ec16f6b) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x59000: op_file_update file(ino 16 size 0x1263b mtime 2021-03-08T09:54:11.078081+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.388 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_file_update file(ino 22 size 0x0 mtime 2021-03-08T09:54:12.770693+0000 allocated 0 extents []) , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_dir_link db/OPTIONS-000013.dbtmp to 22 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _replay 0x6a000: op_file_update file(ino 22 size 0x14ad mtime 2021-03-08T09:54:12.770777+0000 allocated 10000 extents [1:0x4e0000~10000]) 2021-03-08T10:01:42.388 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read left 0x89000 len 0x1000 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.651+0000 7fcd2e001dc0 10 bluefs _replay 0x77000: txn(seq 120 len 0x4c crc 0xb16f6660) 2021-03-08T10:01:42.389 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 10 bluefs _replay 0x84000: txn(seq 133 len 0x23 crc 0xb910e119) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 20 bluefs _replay 0x84000: op_file_update file(ino 36 size 0x61f mtime 2021-03-08T09:54:17.675013+0000 allocated 10000 extents [1:0x4d0000~10000]) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54ba50 0x85000~1000 from file(ino 1 size 0x85000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.390 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.use_adaptive_mutex: 0 , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.rate_limiter: (nil) , 2021-03-08T09:54:19.652+0000 7fcd2e001dc0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-08T10:01:42.391 WARNING:scrape:Ignoring malparsed backtrace: whole_key_filtering: 1 , verify_compression: 0 , read_amp_bytes_per_bit: 0 2021-03-08T10:01:42.393 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 10 bluefs _read_random h 0x7fcd04005050 0x188~2b8 from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 20 bluefs _read_random left 0xe78 0x188~2b8 , 2021-03-08T09:54:19.652+0000 7fcd0e7fc700 20 bluefs _read_random got 696 2021-03-08T10:01:42.394 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 10 bluefs _close_writer 0x55fb4a49c4d0 type 0 , 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 20 bluefs _stop_alloc , 2021-03-08T09:54:19.653+0000 7fcd2e001dc0 1 bdev(0x55fb4a542a80 /var/lib/ceph/osd/ceph-0/block) close 2021-03-08T10:01:42.395 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 20 bluefs _replay 0xa000: op_dir_link db/000005.dbtmp to 10 , 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.914+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.396 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.397 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _replay 0x2c000: txn(seq 45 len 0x24 crc 0x5f1d3bb0) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.397 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x40000: op_file_update file(ino 16 size 0xd7e4 mtime 2021-03-08T09:54:03.076199+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0xbf000 len 0x1000 2021-03-08T10:01:42.398 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x55000~1000 from file(ino 1 size 0x55000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.399 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 10 bluefs _read h 0x55fb4a54df10 0x68000~1000 from file(ino 1 size 0x68000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.399 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_file_remove 23 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x74000: op_file_remove 18 2021-03-08T10:01:42.400 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_file_remove 31 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 20 bluefs _replay 0x81000: op_file_remove 29 2021-03-08T10:01:42.401 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.info_log: 0x55fb4a47f510 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.max_file_opening_threads: 16 , 2021-03-08T09:54:19.915+0000 7fcd2e001dc0 4 rocksdb: Options.statistics: (nil) 2021-03-08T10:01:42.401 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x10 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read fetching 0x0~1000 of 1:0x4b0000~10000 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 20 bluefs _read left 0x1000 len 0x10 2021-03-08T10:01:42.403 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-08T10:01:42.404 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _fsync 0x55fb4a4a43f0 file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _flush 0x55fb4a4a43f0 0x0~93d to file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) , 2021-03-08T09:54:19.916+0000 7fcd2e001dc0 10 bluefs _flush_range 0x55fb4a4a43f0 pos 0x0 0x0~93d to file(ino 38 size 0x0 mtime 2021-03-08T09:54:19.917790+0000 allocated 0 extents []) 2021-03-08T10:01:42.405 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 10 HybridAllocator allocate want 0x10000 unit 0x10000 max_alloc_size 0x10000 hint 0x0 , 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 20 bluefs _flush_range dirty_seq = 136 (was clean) , 2021-03-08T09:54:19.917+0000 7fcd2e001dc0 20 bluefs _flush_range file now file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) 2021-03-08T10:01:42.406 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range file now file(ino 1 size 0x8b000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range in 1:0x10000~400000 x_off 0x8a000 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _flush_range h 0x55fb4a49c4d0 pos now 0x8b000 2021-03-08T10:01:42.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 10 bluefs _read_random h 0x7fccf8004c80 0x0~107b from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluefs _read_random read random 0x0~107b of 1:0x4a0000~10000 2021-03-08T10:01:42.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:19.918+0000 7fcd0e7fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_finalize_thread sleep , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55fb4a56ce60 , 2021-03-08T09:54:19.918+0000 7fcd2e001dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all done 2021-03-08T10:01:42.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment do_bluefs = 1 , 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment initializing bluefs , 2021-03-08T09:54:20.489+0000 7ff636d71dc0 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block 0 2021-03-08T10:01:42.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_link db/CURRENT to 10 , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_unlink db/000005.dbtmp , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0xc000~1000 from file(ino 1 size 0xc000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.490+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x1a000~1000 from file(ino 1 size 0x1a000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _read left 0xe6000 len 0x1000 , 2021-03-08T09:54:20.490+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0xd2000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x2e000: txn(seq 47 len 0x24 crc 0xbc743dc4) 2021-03-08T10:01:42.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x42000: txn(seq 67 len 0x24 crc 0x851bc9b8) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x42000: op_file_update file(ino 16 size 0xd830 mtime 2021-03-08T09:54:03.077262+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x56000: txn(seq 87 len 0x30 crc 0xb9858be4) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x56000: op_file_update file(ino 16 size 0x125a3 mtime 2021-03-08T09:54:11.077692+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x57000~1000 from file(ino 1 size 0x57000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x69000~1000 from file(ino 1 size 0x69000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0x97000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read left 0x8b000 len 0x1000 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _replay 0x75000: txn(seq 118 len 0x5b crc 0xf271c68f) 2021-03-08T10:01:42.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_dir_link db/OPTIONS-000025.dbtmp to 37 , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_file_update file(ino 37 size 0x14ad mtime 2021-03-08T09:54:17.673483+0000 allocated 10000 extents [1:0x4c0000~10000]) , 2021-03-08T09:54:20.491+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca2a910 0x83000~1000 from file(ino 1 size 0x83000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 20 bluefs _read got 279 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 5 rocksdb: verify_sharding column families from rocksdb: [default] , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 10 bluefs open_for_read sharding/recreate_columns (sequential) 2021-03-08T10:01:42.418 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.stats_history_buffer_size: 1048576 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.max_open_files: -1 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.bytes_per_sync: 0 2021-03-08T10:01:42.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.target_file_size_multiplier: 1 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 , 2021-03-08T09:54:20.492+0000 7ff636d71dc0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2021-03-08T10:01:42.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.493+0000 7ff6167fc700 10 bluefs _read_random h 0x7ff600001d20 0x11a~4a from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:20.493+0000 7ff6167fc700 20 bluefs _read_random left 0xee6 0x11a~4a , 2021-03-08T09:54:20.493+0000 7ff6167fc700 20 bluefs _read_random got 74 2021-03-08T10:01:42.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) _prepare_db_environment initializing bluefs , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block 0 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 1 bdev(0x5646fca28a30 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block 2021-03-08T10:01:42.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _replay 0xb000: op_dir_unlink db/000005.dbtmp , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0xc000~1000 from file(ino 1 size 0xc000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read left 0xf4000 len 0x1000 2021-03-08T10:01:42.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read left 0xe6000 len 0x1000 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.757+0000 7ff636d71dc0 10 bluefs _replay 0x1a000: txn(seq 27 len 0x24 crc 0x91ca6867) 2021-03-08T10:01:42.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x2e000: txn(seq 47 len 0x24 crc 0xbc743dc4) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x2e000: op_file_update file(ino 16 size 0x9e52 mtime 2021-03-08T09:54:03.064379+0000 allocated 10000 extents [1:0x480000~10000]) 2021-03-08T10:01:42.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x42000: txn(seq 67 len 0x24 crc 0x851bc9b8) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x42000: op_file_update file(ino 16 size 0xd830 mtime 2021-03-08T09:54:03.077262+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x43000~1000 from file(ino 1 size 0x43000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x56000: op_file_update file(ino 16 size 0x125a3 mtime 2021-03-08T09:54:11.077692+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x57000~1000 from file(ino 1 size 0x57000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0xa9000 len 0x1000 2021-03-08T10:01:42.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0x97000 len 0x1000 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x69000: txn(seq 106 len 0x7c crc 0x67e69dad) 2021-03-08T10:01:42.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _replay 0x75000: txn(seq 118 len 0x5b crc 0xf271c68f) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x75000: op_file_update file(ino 29 size 0x0 mtime 2021-03-08T09:54:16.033410+0000 allocated 0 extents []) 2021-03-08T10:01:42.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _replay 0x82000: op_file_update file(ino 37 size 0x14ad mtime 2021-03-08T09:54:17.673483+0000 allocated 10000 extents [1:0x4c0000~10000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac5010 0x83000~1000 from file(ino 1 size 0x83000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:20.758+0000 7ff636d71dc0 20 bluefs _read left 0x7d000 len 0x1000 2021-03-08T10:01:42.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 20 bluefs _read got 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 10 bluefs open_for_read db/MANIFEST-000026 (sequential) 2021-03-08T10:01:42.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:20.759+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-08T10:01:42.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 10 bluefs _read_random h 0x7ff600003ab0 0x58c~4a from file(ino 38 size 0x93d mtime 2021-03-08T09:54:19.917885+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 20 bluefs _read_random left 0xa74 0x58c~4a , 2021-03-08T09:54:20.759+0000 7ff5f6ffd700 20 bluefs _read_random got 74 2021-03-08T10:01:42.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow 0x0~a59: 0x0~a59 Blob(0x5646fca2c190 blob([0xf000~1000] csum crc32c/0x1000) use_tracker(0x1000 0xa59) SharedBlob(0x5646fcac6970 sbid 0x0)) , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore.sharedblob(0x5646fcac6970) put 0x5646fcac6970 removing self from set 0x5646fcac6570 , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore.BufferSpace(0x5646fcac6988 in 0x5646fc98ba10) _clear 2021-03-08T10:01:42.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:20.763+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection 1.0_head cnode(bits 0) , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:00000000::::head# , 2021-03-08T09:54:20.763+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:d4465517:::INTEL_SSDPED1D480GA_PHMB7512009A480DGN:head# 2021-03-08T10:01:42.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.254+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:21.254+0000 7ff636d71dc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:21.254+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:21.255+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac7c60 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 20 bluefs _read got 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 10 bluefs open_for_read db/MANIFEST-000026 (sequential) , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 10 bluefs open_for_read h 0x5646fca12750 on file(ino 39 size 0x117 mtime 2021-03-08T09:54:19.918263+0000 allocated 10000 extents [1:0x430000~10000]) 2021-03-08T10:01:42.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-08T10:01:42.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 , 2021-03-08T09:54:21.256+0000 7ff636d71dc0 4 rocksdb: Options.compression_opts.enabled: false 2021-03-08T10:01:42.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.256+0000 7ff6167fc700 20 bluefs _read_random got 85 , 2021-03-08T09:54:21.256+0000 7ff6167fc700 10 bluefs _read_random h 0x7ff604004aa0 0x188~2b8 from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:21.256+0000 7ff6167fc700 20 bluefs _read_random left 0xe78 0x188~2b8 2021-03-08T10:01:42.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.257+0000 7ff636d71dc0 1 bdev(0x5646fca28a30 /var/lib/ceph/osd/ceph-0/block) close , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) 2021-03-08T10:01:42.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xf5000 len 0x1000 , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xe7000 len 0x1000 2021-03-08T10:01:42.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.518+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read left 0xd3000 len 0x1000 , 2021-03-08T09:54:21.518+0000 7ff636d71dc0 20 bluefs _read got 4096 2021-03-08T10:01:42.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x41000: txn(seq 66 len 0x24 crc 0x5306814b) 2021-03-08T10:01:42.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x55000: op_file_update file(ino 16 size 0x1257d mtime 2021-03-08T09:54:11.077559+0000 allocated 20000 extents [1:0x480000~10000,1:0x4a0000~10000]) 2021-03-08T10:01:42.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read got 4096 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x68000: op_dir_unlink db/CURRENT 2021-03-08T10:01:42.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x74000: op_file_remove 13 2021-03-08T10:01:42.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _read h 0x5646fcac2cc0 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read left 0x7e000 len 0x1000 2021-03-08T10:01:42.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs open_for_read h 0x5646fca0e110 on file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 10 bluefs _read h 0x5646fca0e110 0x0~2000 from file(ino 40 size 0x10 mtime 2021-03-08T09:54:19.918425+0000 allocated 10000 extents [1:0x460000~10000]) , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 20 bluefs _read reaching (or past) eof, len clipped to 0x10 2021-03-08T10:01:42.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.atomic_flush: 0 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 , 2021-03-08T09:54:21.519+0000 7ff636d71dc0 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-08T10:01:42.449 WARNING:scrape:Ignoring malparsed backtrace: verify_compression: 0 , read_amp_bytes_per_bit: 0 , format_version: 2 2021-03-08T10:01:42.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.520+0000 7ff5f6ffd700 20 bluefs _read_random got 697 , 2021-03-08T09:54:21.520+0000 7ff616ffd700 20 bluefs _read left 0x1000 len 0x4ca , 2021-03-08T09:54:21.520+0000 7ff616ffd700 20 bluefs _read got 1226 2021-03-08T10:01:42.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 bluefs _claim_completed_aios got 1 aios , 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 wait_for_aio 0x5646fc9824d0 , 2021-03-08T09:54:21.520+0000 7ff636d71dc0 10 bluefs wait_for_aio 0x5646fc9824d0 done in 0.000019 2021-03-08T10:01:42.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs _flush_range in 1:0x4b0000~10000 x_off 0x0 , 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs flush_buffer caching tail of 0x4ad and padding block with 0xb53 buffer.length() 0 , 2021-03-08T09:54:21.521+0000 7ff636d71dc0 20 bluefs _flush_range h 0x5646fc980510 pos now 0x14ad 2021-03-08T10:01:42.452 WARNING:scrape:Ignoring malparsed backtrace: Flush(GB): cumulative 0.000, interval 0.000 , AddFile(GB): cumulative 0.000, interval 0.000 , AddFile(Total Files): cumulative 0, interval 0 2021-03-08T10:01:42.453 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:21.522+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x5646fca2a830 , 2021-03-08T09:54:21.522+0000 7ff636d71dc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x5646fca32650 , 2021-03-08T09:54:21.522+0000 7ff5fd7fa700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_sync_thread sleep 2021-03-08T10:01:42.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.631+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:22.631+0000 7fb50e40adc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:22.631+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.455 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.632+0000 7fb50e40adc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.458 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.458 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.459 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _read h 0x55eec084b910 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.460 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 10 bluefs _replay 0x8f000: txn(seq 144 len 0x7c crc 0x97e17074) , 2021-03-08T09:54:22.633+0000 7fb50e40adc0 20 bluefs _replay 0x8f000: op_file_update file(ino 45 size 0x0 mtime 2021-03-08T09:54:21.522082+0000 allocated 0 extents []) 2021-03-08T10:01:42.461 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:22.634+0000 7fb50e40adc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.462 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.462 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs open_for_read db/000022.sst (random) , 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs open_for_read h 0x7fb4dc001d20 on file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:22.634+0000 7fb4f67fc700 10 bluefs _read h 0x7fb4dc001d20 0x0~4ca from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) prefetch 2021-03-08T10:01:42.463 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.635+0000 7fb50e40adc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:22.635+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4d4001d20 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:22.635+0000 7fb50e40adc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 2021-03-08T10:01:42.464 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.895+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.465 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.895+0000 7fb50e40adc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) 2021-03-08T10:01:42.465 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.466 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xc8000 len 0x1000 2021-03-08T10:01:42.467 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0854350 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.468 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) 2021-03-08T10:01:42.469 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) 2021-03-08T10:01:42.470 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) 2021-03-08T10:01:42.471 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs _replay 0x89000: txn(seq 138 len 0x7c crc 0x140649b0) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_file_update file(ino 41 size 0x0 mtime 2021-03-08T09:54:19.918678+0000 allocated 0 extents []) , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_dir_link db.wal/000027.log to 41 2021-03-08T10:01:42.471 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 20 bluefs _read got 0 , 2021-03-08T09:54:22.896+0000 7fb50e40adc0 10 bluefs open_for_read db/MANIFEST-000029 (sequential) 2021-03-08T10:01:42.472 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.473 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.strategy: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 , 2021-03-08T09:54:22.897+0000 7fb50e40adc0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-08T10:01:42.473 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.897+0000 7fb4f67fc700 20 bluefs _read_random left 0xa06 0x5fa~2b9 , 2021-03-08T09:54:22.897+0000 7fb4f67fc700 20 bluefs _read_random got 697 , 2021-03-08T09:54:22.897+0000 7fb4ca7fc700 10 bluefs _read_random h 0x7fb4e0003ac0 0x11a~4a from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) 2021-03-08T10:01:42.474 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow 0x0~a59: 0x0~a59 Blob(0x55eec084d190 blob([0xf000~1000] csum crc32c/0x1000) use_tracker(0x1000 0xa59) SharedBlob(0x55eec0863c00 sbid 0x0)) , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore.sharedblob(0x55eec0863c00) put 0x55eec0863c00 removing self from set 0x55eec085e460 , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore.BufferSpace(0x55eec0863c18 in 0x55eec07aca10) _clear 2021-03-08T10:01:42.475 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:22.901+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _fsck_check_objects collection 1.0_head cnode(bits 0) , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:00000000::::head# , 2021-03-08T09:54:22.901+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0) fsck_check_objects_shallow #1:d4465517:::INTEL_SSDPED1D480GA_PHMB7512009A480DGN:head# 2021-03-08T10:01:42.476 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.399+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label got bdev(osd_uuid 950fedc6-c9ca-43a9-b7c0-5778c6d984d6, size 0x1680000000, btime 2021-03-08T09:53:37.915628+0000, desc main, 9 meta) , 2021-03-08T09:54:23.399+0000 7fb50e40adc0 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 , 2021-03-08T09:54:23.399+0000 7fb50e40adc0 10 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label 2021-03-08T10:01:42.476 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0xa000: op_file_update file(ino 10 size 0x10 mtime 2021-03-08T09:53:40.232871+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0xb000~1000 from file(ino 1 size 0xb000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xf5000 len 0x1000 2021-03-08T10:01:42.477 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x18000: txn(seq 25 len 0x24 crc 0x69863074) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x18000: op_file_update file(ino 16 size 0x482e mtime 2021-03-08T09:53:59.055154+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x19000~1000 from file(ino 1 size 0x19000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.478 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x2c000: op_file_update file(ino 16 size 0x8cbf mtime 2021-03-08T09:54:02.078553+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x2d000~1000 from file(ino 1 size 0x2d000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xd3000 len 0x1000 2021-03-08T10:01:42.479 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x41000~1000 from file(ino 1 size 0x41000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xbf000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.479 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0xab000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x55000: txn(seq 86 len 0x30 crc 0xb7e0d732) 2021-03-08T10:01:42.480 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read left 0x98000 len 0x1000 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _replay 0x68000: txn(seq 105 len 0x4c crc 0xa0607ca4) 2021-03-08T10:01:42.480 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000010.sst , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_file_remove 18 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x74000: op_dir_unlink db/000007.sst 2021-03-08T10:01:42.481 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_dir_unlink db/MANIFEST-000020 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 20 bluefs _replay 0x81000: op_file_remove 29 , 2021-03-08T09:54:23.400+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0858ef0 0x82000~1000 from file(ino 1 size 0x82000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.482 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 10 bluefs _replay 0x8f000: txn(seq 144 len 0x7c crc 0x97e17074) , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 20 bluefs _replay 0x8f000: op_file_update file(ino 45 size 0x0 mtime 2021-03-08T09:54:21.522082+0000 allocated 0 extents []) 2021-03-08T10:01:42.482 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_reads: 0 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.allow_mmap_writes: 0 , 2021-03-08T09:54:23.401+0000 7fb50e40adc0 4 rocksdb: Options.use_direct_reads: 0 2021-03-08T10:01:42.483 WARNING:scrape:Ignoring malparsed backtrace: data_block_hash_table_util_ratio: 0.750000 , hash_index_allow_collision: 1 , checksum: 1 2021-03-08T10:01:42.484 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs open_for_read db/000022.sst (random) , 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs open_for_read h 0x7fb4d4004c60 on file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) , 2021-03-08T09:54:23.401+0000 7fb4f67fc700 10 bluefs _read h 0x7fb4d4004c60 0x0~4ca from file(ino 33 size 0x4ca mtime 2021-03-08T09:54:17.672491+0000 allocated 10000 extents [1:0x470000~10000]) prefetch 2021-03-08T10:01:42.484 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.402+0000 7fb50e40adc0 20 bluefs _read_random got 140 , 2021-03-08T09:54:23.402+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4e00039c0 0x2210~ecf from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:23.402+0000 7fb50e40adc0 20 bluefs _read_random read random 0x2210~ecf of 1:0x4a0000~10000 2021-03-08T10:01:42.485 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.663+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x4000~1000 from file(ino 1 size 0x4000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read left 0xfc000 len 0x1000 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.486 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read left 0xee000 len 0x1000 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.663+0000 7fb50e40adc0 10 bluefs _replay 0x12000: txn(seq 19 len 0x4c crc 0x860758f9) 2021-03-08T10:01:42.487 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x23000: txn(seq 36 len 0x24 crc 0xb42c543) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x23000: op_file_update file(ino 16 size 0x8196 mtime 2021-03-08T09:54:02.067946+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x24000~1000 from file(ino 1 size 0x24000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) 2021-03-08T10:01:42.488 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x37000: op_file_update file(ino 16 size 0xa867 mtime 2021-03-08T09:54:03.071229+0000 allocated 10000 extents [1:0x480000~10000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x38000~1000 from file(ino 1 size 0x38000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xc8000 len 0x1000 2021-03-08T10:01:42.490 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _read h 0x55eec0856160 0x4c000~1000 from file(ino 1 size 0x4c000 mtime 0.000000 allocated 400000 extents [1:0x10000~400000]) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xb4000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 2021-03-08T10:01:42.491 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0xa0000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x60000: txn(seq 97 len 0x30 crc 0x24a82ade) 2021-03-08T10:01:42.492 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0x91000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x6f000: txn(seq 112 len 0x4c crc 0xfe4cffab) 2021-03-08T10:01:42.493 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read left 0x85000 len 0x1000 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 4096 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x7b000: txn(seq 124 len 0x23 crc 0xb48f7f0f) 2021-03-08T10:01:42.494 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs _replay 0x89000: txn(seq 138 len 0x7c crc 0x140649b0) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_file_update file(ino 41 size 0x0 mtime 2021-03-08T09:54:19.918678+0000 allocated 0 extents []) , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _replay 0x89000: op_dir_link db.wal/000027.log to 41 2021-03-08T10:01:42.495 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read reaching (or past) eof, len clipped to 0x0 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 20 bluefs _read got 0 , 2021-03-08T09:54:23.664+0000 7fb50e40adc0 10 bluefs open_for_read db/MANIFEST-000029 (sequential) 2021-03-08T10:01:42.496 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.max_background_compactions: 2 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.writable_file_max_buffer_size: 0 2021-03-08T10:01:42.497 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.max_write_buffer_number: 4 , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.compression: NoCompression , 2021-03-08T09:54:23.665+0000 7fb50e40adc0 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-08T10:01:42.497 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.665+0000 7fb4f5ffb700 20 bluefs _read_random got 53 , 2021-03-08T09:54:23.665+0000 7fb4f67fc700 10 bluefs _read_random h 0x7fb4d4005cd0 0x908~35 from file(ino 38 size 0x93d mtime 2021-03-08T09:54:19.917885+0000 allocated 10000 extents [1:0x420000~10000]) , 2021-03-08T09:54:23.665+0000 7fb4f67fc700 20 bluefs _read_random left 0x6f8 0x908~35 2021-03-08T10:01:42.498 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.666+0000 7fb50e40adc0 10 bluefs open_for_write db/000032.dbtmp , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs open_for_write mapping db/000032.dbtmp vsel_hint 0x1 , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 10 bluefs open_for_write h 0x55eec0879010 on file(ino 48 size 0x0 mtime 2021-03-08T09:54:23.667013+0000 allocated 0 extents []) 2021-03-08T10:01:42.499 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _fsync file metadata was dirty (151) on file(ino 50 size 0x14ad mtime 2021-03-08T09:54:23.667558+0000 allocated 10000 extents [1:0x490000~10000]), flushing log , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _flush_and_sync_log 1 dirty_files , 2021-03-08T09:54:23.666+0000 7fb50e40adc0 20 bluefs _flush_and_sync_log op_file_update file(ino 50 size 0x14ad mtime 2021-03-08T09:54:23.667558+0000 allocated 10000 extents [1:0x490000~10000]) 2021-03-08T10:01:42.499 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluefs _read_random got 282 , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 10 bluefs _read_random h 0x7fb4e0003ba0 0x41c7~8c from file(ino 28 size 0x45b3 mtime 2021-03-08T09:54:14.369778+0000 allocated 10000 extents [1:0x4a0000~10000]) , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluefs _read_random left 0xe39 0x41c7~8c 2021-03-08T10:01:42.500 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55eec085de00 , 2021-03-08T09:54:23.667+0000 7fb4f67fc700 20 bluestore(/var/lib/ceph/osd/ceph-0) _kv_finalize_thread sleep , 2021-03-08T09:54:23.667+0000 7fb50e40adc0 20 bluestore(/var/lib/ceph/osd/ceph-0) _osr_drain_all drain 0x55eec085e1b0 2021-03-08T10:01:42.500 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:54:34.490 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] , 2021-03-08T09:54:39.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2021-03-08T10:01:42.501 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T09:56:47.941 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2021-03-08T09:56:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-08T10:01:42.502 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2021-03-08T09:58:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2021-03-08T09:58:56.167 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-03-08T10:01:42.503 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/ceph_manager.py", line 173, in wrapper , return func(self) , File "/home/teuthworker/src/github.com_ceph_ceph_ad59f0144084ddad5a804bf51eca43873e5a1af9/qa/tasks/ceph_manager.py", line 1261, in _do_thrash 2021-03-08T10:01:42.504 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:00:48.076 INFO:teuthology.orchestra.run.smithi046.stdout:{ , 2021-03-08T10:00:48.077 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "" , 2021-03-08T10:00:48.077 INFO:teuthology.orchestra.run.smithi046.stdout:} 2021-03-08T10:01:42.505 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:01:10.739 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com , 2021-03-08T10:01:10.740 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2021-03-08T10:01:10.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2021-03-08T10:01:42.506 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-08T10:01:12.932 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-03-08T10:01:12.933 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com , 2021-03-08T10:01:12.933 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-03-08T10:01:42.516 INFO:scrape:Found 1 distinct failure reasons 2021-03-08T10:01:42.518 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2021-03-08T10:01:42.518 INFO:scrape:1 jobs: ['5946394'] 2021-03-08T10:01:42.519 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2021-03-08T10:01:42.519 INFO:scrape: 2021-03-08T10:01:42.520 INFO:teuthology.results:Sending results to xxhdx1985126@gmail.com: Test Run: xuxuehan-2021-03-08_09:35:35-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/xuxuehan-2021-03-08_09:35:35-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/xuxuehan-2021-03-08_09:35:35-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5946394] crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:14:37 info: http://pulpito.ceph.com/xuxuehan-2021-03-08_09:35:35-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/5946394/ log: http://qa-proxy.ceph.com/teuthology/xuxuehan-2021-03-08_09:35:35-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/5946394/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'