2020-10-22T12:04:22.181 INFO:root:teuthology version: 1.0.0-8c6604d 2020-10-22T12:04:22.183 INFO:teuthology.results:subset: '421/2057' 2020-10-22T12:04:22.183 INFO:teuthology.results:seed: '197' 2020-10-22T12:04:30.340 INFO:root:teuthology version: 1.0.0-8c6604d 2020-10-22T12:04:30.341 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-22T12:42:31.012 INFO:teuthology.results:Tests finished! gathering results... 2020-10-22T12:42:31.121 INFO:scrape:Found 1 jobs 2020-10-22T12:42:31.262 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.11-85-g200fda8bb9 (200fda8bb95fcd626ae7aef62151966fb83c6c48) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7fa4543dc8d4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5be) [0x55af08a6c14e] 2020-10-22T12:42:31.263 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.318361", 2020-10-22T12:42:31.263 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { , 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", , 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2020-10-22T12:42:31.264 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.324862", , 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg" , 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.264 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4314.0:10 2.es0 2:798170f8:::smithi08938662-4:head [append 0~50421,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-10-22T12:42:31.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.160 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.327417", 2020-10-22T12:42:31.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] , 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.319697", , 2020-10-22T12:24:20.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg" , 2020-10-22T12:24:20.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done" , 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2020-10-22T12:42:31.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.368996", 2020-10-22T12:42:31.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.406670", , 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_committed" 2020-10-22T12:42:31.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:20.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.268 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.029201202999999998, , 2020-10-22T12:24:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { , 2020-10-22T12:24:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2020-10-22T12:42:31.268 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.271 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.517485", , 2020-10-22T12:24:20.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg" 2020-10-22T12:42:31.269 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.548921", , 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done" , 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2020-10-22T12:42:31.269 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.326558", , 2020-10-22T12:24:20.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched" , 2020-10-22T12:24:20.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.270 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.367087", , 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched" 2020-10-22T12:42:31.270 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.378731", 2020-10-22T12:42:31.271 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" , 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.271 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.389624", , 2020-10-22T12:24:20.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" , 2020-10-22T12:24:20.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.272 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.473124", , 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" 2020-10-22T12:42:31.272 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.489919", 2020-10-22T12:42:31.273 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:21.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:21.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.273 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.535787", , 2020-10-22T12:24:21.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:21.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.274 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.536160", , 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" 2020-10-22T12:42:31.274 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.769 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "done" , 2020-10-22T12:24:21.770 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.770 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2020-10-22T12:42:31.275 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.313456", 2020-10-22T12:42:31.275 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.330772", , 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" 2020-10-22T12:42:31.275 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.276 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.831 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "header_read" , 2020-10-22T12:24:21.831 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.832 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.276 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpWrite(2.ds1 25/22 ECSubWrite(tid=12, reqid=client.4314.0:49, at_version=25'4, trim_to=0'0, roll_forward_to=24'1))", , 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.340811", , 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 2.4027936730000001, 2020-10-22T12:42:31.277 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpWrite(2.ds1 25/22 ECSubWrite(tid=20, reqid=client.4314.0:52, at_version=25'6, trim_to=0'0, roll_forward_to=25'3))", , 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.344658", 2020-10-22T12:42:31.277 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.876 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.398794", , 2020-10-22T12:24:21.877 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_committed" , 2020-10-22T12:24:21.877 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.891 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "initiated" , 2020-10-22T12:24:21.891 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.892 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.906 INFO:teuthology.orchestra.run.smithi057.stdout: ] , 2020-10-22T12:24:21.906 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: ] , 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: } 2020-10-22T12:42:31.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.508271", , 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.280 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.779 INFO:teuthology.orchestra.run.smithi057.stdout:{ , 2020-10-22T12:24:22.780 INFO:teuthology.orchestra.run.smithi057.stdout: "size": 20, , 2020-10-22T12:24:22.780 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 600, 2020-10-22T12:42:31.280 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.305767", , 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: "events": [ , 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.394952", 2020-10-22T12:42:31.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.393377", , 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" 2020-10-22T12:42:31.295 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "reached pg", , 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.296 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.406584", 2020-10-22T12:42:31.296 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.893 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.433674", , 2020-10-22T12:24:22.894 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" , 2020-10-22T12:24:22.894 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.297 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 3.3210408409999999, , 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 0.025615425000000001, , 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "type_data": { 2020-10-22T12:42:31.297 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.438786", 2020-10-22T12:42:31.298 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: "client_addr": "172.21.15.89:0/1642317874", , 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: "tid": 117 , 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.298 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.957 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.957 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.958 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpReadReply(2.3s0 25/22 ECSubReadReply(tid=22, attrs_read=0))", 2020-10-22T12:42:31.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.311152", 2020-10-22T12:42:31.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:23.974 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:23.975 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:23.975 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.015 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.314272", , 2020-10-22T12:24:24.015 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:24.016 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.324831", , 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" 2020-10-22T12:42:31.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.342268", 2020-10-22T12:42:31.301 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_committed" , 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.301 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.403405", , 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "started" 2020-10-22T12:42:31.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "started", , 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.128 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.129 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.129 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.415018", 2020-10-22T12:42:31.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.446952", , 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 4.4306505979999997, , 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 0.036566672000000001, , 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "type_data": { 2020-10-22T12:42:31.304 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.297809", 2020-10-22T12:42:31.304 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.324731", , 2020-10-22T12:24:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" 2020-10-22T12:42:31.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "started", , 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "osd_op(client.4314.0:56 2.0s0 2:06f3166f:::smithi08938662-19 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [setxattr _header (357)] snapc 0=[] ondisk+write+known_if_redirected e25)", , 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.337798", 2020-10-22T12:42:31.307 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.338020", , 2020-10-22T12:24:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "dispatched" 2020-10-22T12:42:31.307 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.389364", , 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "commit_sent" , 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "dispatched" , 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.462113", 2020-10-22T12:42:31.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.426748", , 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" 2020-10-22T12:42:31.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.463685", , 2020-10-22T12:24:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "throttled" , 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.310 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.497733", 2020-10-22T12:42:31.310 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.524406", , 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "throttled" , 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:50.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574865 got 55834574865 for osd.2 , 2020-10-22T12:24:50.119 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.3 , 2020-10-22T12:24:50.631 INFO:teuthology.orchestra.run.smithi089.stdout:55834574864 2020-10-22T12:42:31.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.047 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6789/0", , 2020-10-22T12:24:55.048 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6789/0" , 2020-10-22T12:24:55.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.313 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2020-10-22T12:42:31.313 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.313 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.090 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.090 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", , 2020-10-22T12:24:55.091 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-10-22T12:42:31.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] , 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.121 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, , 2020-10-22T12:24:55.122 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2020-10-22T12:42:31.315 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_mirroring": { , 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_mirroring", 2020-10-22T12:42:31.315 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.316 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.316 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.178 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.178 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", , 2020-10-22T12:24:55.179 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.317 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] , 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.317 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.318 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { , 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", , 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2020-10-22T12:42:31.318 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.240 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.241 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.241 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.319 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.255 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.256 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.319 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.320 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { , 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", , 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2020-10-22T12:42:31.320 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.300 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2020-10-22T12:42:31.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.328 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.329 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.329 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.322 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { , 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", , 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2020-10-22T12:42:31.322 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.322 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2020-10-22T12:42:31.323 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.323 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", , 2020-10-22T12:24:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.324 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.324 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2020-10-22T12:42:31.325 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2020-10-22T12:42:31.325 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", , 2020-10-22T12:24:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", , 2020-10-22T12:24:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2020-10-22T12:42:31.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", , 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2020-10-22T12:42:31.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { , 2020-10-22T12:24:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2020-10-22T12:42:31.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", , 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", , 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.330 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "localhost", , 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.330 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.331 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.332 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", , 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.333 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, , 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", , 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.333 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { , 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", , 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2020-10-22T12:42:31.334 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", , 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.334 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.684 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", , 2020-10-22T12:24:55.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.335 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", , 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.335 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", , 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2020-10-22T12:42:31.336 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.726 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", , 2020-10-22T12:24:55.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.337 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: ], , 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: "services": {}, , 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: "always_on_modules": { 2020-10-22T12:42:31.338 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:25:08.182 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:25:08.183 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:25:08.183 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-22 12:24:22.295240 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.340 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:26:10.194 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:10.194 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:10.195 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.341 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:26:51.581 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2020-10-22 12:24:19.867927 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=19, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:51.581 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:27:11.582 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.342 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:27:31.719 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.f , instructing pg 1.f on osd.6 to deep-scrub , 2020-10-22T12:27:32.343 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.10 2020-10-22T12:42:31.343 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:28:35.645 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:28:35.645 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:28:35.646 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-10-22 12:24:20.879074 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.344 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:29:37.641 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:29:37.642 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:29:37.642 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.345 WARNING:scrape:Ignoring malparsed backtrace: instructing pg 1.11 on osd.0 to deep-scrub , 2020-10-22T12:29:58.966 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:30:18.968 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.347 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.348 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:32:02.940 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:02.940 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:02.941 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.349 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:32:43.699 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:43.700 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:43.700 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.350 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:33:25.035 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-22 12:24:29.921190 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:33:25.035 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:33:25.036 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-22 12:24:22.839282 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.351 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:34:26.998 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:34:26.999 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:34:26.999 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.352 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:35:08.808 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:08.808 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:08.809 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.354 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:35:50.124 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2020-10-22 12:24:19.867927 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=19, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:50.127 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:36:10.128 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.354 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:36:52.155 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:36:52.156 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:36:52.157 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.356 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:37:33.738 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:37:33.738 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:37:33.739 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-10-22 12:24:20.879074 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.357 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:38:35.692 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:38:35.693 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:38:35.693 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.358 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-22 12:24:29.921190 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-22 12:24:22.839282 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.359 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-22 12:24:22.295240 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.360 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:00.295 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:41:00.296 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:41:00.296 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.361 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:20.929 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-22T12:41:20.966 INFO:tasks.ceph.osd.0:Stopped , 2020-10-22T12:41:20.967 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-10-22T12:42:31.361 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:34.141 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1022/qa/tasks/ceph.py", line 1873, in task 2020-10-22T12:42:31.362 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:35.575 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-22T12:41:35.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com , 2020-10-22T12:41:35.577 INFO:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2020-10-22T12:42:31.393 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.11-85-g200fda8bb9 (200fda8bb95fcd626ae7aef62151966fb83c6c48) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7fa4543dc8d4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5be) [0x55af08a6c14e] 2020-10-22T12:42:31.394 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.318361", 2020-10-22T12:42:31.395 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { , 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", , 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2020-10-22T12:42:31.395 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.324862", , 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg" , 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.396 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4314.0:10 2.es0 2:798170f8:::smithi08938662-4:head [append 0~50421,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-10-22T12:42:31.396 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.160 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.327417", 2020-10-22T12:42:31.397 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] , 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.397 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.319697", , 2020-10-22T12:24:20.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg" , 2020-10-22T12:24:20.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.398 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done" , 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2020-10-22T12:42:31.398 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.368996", 2020-10-22T12:42:31.399 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.406670", , 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_committed" 2020-10-22T12:42:31.399 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:20.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.399 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.029201202999999998, , 2020-10-22T12:24:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { , 2020-10-22T12:24:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2020-10-22T12:42:31.400 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.271 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.517485", , 2020-10-22T12:24:20.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg" 2020-10-22T12:42:31.400 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.548921", , 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done" , 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2020-10-22T12:42:31.401 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.326558", , 2020-10-22T12:24:20.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched" , 2020-10-22T12:24:20.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.401 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.367087", , 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched" 2020-10-22T12:42:31.402 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.378731", 2020-10-22T12:42:31.402 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" , 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.403 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.389624", , 2020-10-22T12:24:20.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" , 2020-10-22T12:24:20.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.403 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.473124", , 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" 2020-10-22T12:42:31.403 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.489919", 2020-10-22T12:42:31.404 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:21.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:21.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.404 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.535787", , 2020-10-22T12:24:21.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:21.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.405 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.536160", , 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" 2020-10-22T12:42:31.405 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.769 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "done" , 2020-10-22T12:24:21.770 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.770 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2020-10-22T12:42:31.406 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.313456", 2020-10-22T12:42:31.406 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.330772", , 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" 2020-10-22T12:42:31.406 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.407 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.831 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "header_read" , 2020-10-22T12:24:21.831 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.832 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.407 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpWrite(2.ds1 25/22 ECSubWrite(tid=12, reqid=client.4314.0:49, at_version=25'4, trim_to=0'0, roll_forward_to=24'1))", , 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.340811", , 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 2.4027936730000001, 2020-10-22T12:42:31.408 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpWrite(2.ds1 25/22 ECSubWrite(tid=20, reqid=client.4314.0:52, at_version=25'6, trim_to=0'0, roll_forward_to=25'3))", , 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.344658", 2020-10-22T12:42:31.408 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.876 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.398794", , 2020-10-22T12:24:21.877 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_committed" , 2020-10-22T12:24:21.877 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.409 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.891 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "initiated" , 2020-10-22T12:24:21.891 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.892 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.409 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.906 INFO:teuthology.orchestra.run.smithi057.stdout: ] , 2020-10-22T12:24:21.906 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.410 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: ] , 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: } 2020-10-22T12:42:31.410 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.508271", , 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.411 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.779 INFO:teuthology.orchestra.run.smithi057.stdout:{ , 2020-10-22T12:24:22.780 INFO:teuthology.orchestra.run.smithi057.stdout: "size": 20, , 2020-10-22T12:24:22.780 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 600, 2020-10-22T12:42:31.411 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.305767", , 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.411 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: "events": [ , 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.412 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.394952", 2020-10-22T12:42:31.412 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.393377", , 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" 2020-10-22T12:42:31.413 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "reached pg", , 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.413 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.406584", 2020-10-22T12:42:31.414 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.893 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.433674", , 2020-10-22T12:24:22.894 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" , 2020-10-22T12:24:22.894 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.414 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 3.3210408409999999, , 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 0.025615425000000001, , 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "type_data": { 2020-10-22T12:42:31.415 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.438786", 2020-10-22T12:42:31.415 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: "client_addr": "172.21.15.89:0/1642317874", , 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: "tid": 117 , 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.416 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.957 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.957 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.958 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpReadReply(2.3s0 25/22 ECSubReadReply(tid=22, attrs_read=0))", 2020-10-22T12:42:31.416 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.311152", 2020-10-22T12:42:31.416 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:23.974 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:23.975 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:23.975 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.417 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.015 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.314272", , 2020-10-22T12:24:24.015 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:24.016 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.417 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.324831", , 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" 2020-10-22T12:42:31.418 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.342268", 2020-10-22T12:42:31.418 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_committed" , 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.419 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.403405", , 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "started" 2020-10-22T12:42:31.419 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "started", , 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.420 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.128 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.129 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.129 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.415018", 2020-10-22T12:42:31.420 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.446952", , 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.420 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 4.4306505979999997, , 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 0.036566672000000001, , 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "type_data": { 2020-10-22T12:42:31.421 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.297809", 2020-10-22T12:42:31.421 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.324731", , 2020-10-22T12:24:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" 2020-10-22T12:42:31.422 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "started", , 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.422 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.423 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "osd_op(client.4314.0:56 2.0s0 2:06f3166f:::smithi08938662-19 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [setxattr _header (357)] snapc 0=[] ondisk+write+known_if_redirected e25)", , 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.337798", 2020-10-22T12:42:31.423 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.338020", , 2020-10-22T12:24:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "dispatched" 2020-10-22T12:42:31.424 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.389364", , 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "commit_sent" , 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.424 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "dispatched" , 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.424 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.462113", 2020-10-22T12:42:31.425 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.426748", , 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" 2020-10-22T12:42:31.425 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.463685", , 2020-10-22T12:24:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.426 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "throttled" , 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.426 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.497733", 2020-10-22T12:42:31.427 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.524406", , 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "throttled" , 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.428 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:50.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574865 got 55834574865 for osd.2 , 2020-10-22T12:24:50.119 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.3 , 2020-10-22T12:24:50.631 INFO:teuthology.orchestra.run.smithi089.stdout:55834574864 2020-10-22T12:42:31.428 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.047 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6789/0", , 2020-10-22T12:24:55.048 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6789/0" , 2020-10-22T12:24:55.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.429 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2020-10-22T12:42:31.429 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.430 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.090 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.090 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", , 2020-10-22T12:24:55.091 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-10-22T12:42:31.430 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] , 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.431 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.121 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, , 2020-10-22T12:24:55.122 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2020-10-22T12:42:31.431 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_mirroring": { , 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_mirroring", 2020-10-22T12:42:31.432 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.432 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.432 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.178 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.178 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", , 2020-10-22T12:24:55.179 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.433 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] , 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.433 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.434 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { , 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", , 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2020-10-22T12:42:31.434 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.240 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.241 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.241 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.435 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.255 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.256 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.435 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.436 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { , 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", , 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2020-10-22T12:42:31.436 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.300 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.436 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2020-10-22T12:42:31.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.328 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.329 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.329 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { , 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", , 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2020-10-22T12:42:31.438 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.438 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2020-10-22T12:42:31.439 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.439 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", , 2020-10-22T12:24:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2020-10-22T12:42:31.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2020-10-22T12:42:31.441 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", , 2020-10-22T12:24:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", , 2020-10-22T12:24:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2020-10-22T12:42:31.441 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", , 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.442 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2020-10-22T12:42:31.442 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.443 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { , 2020-10-22T12:24:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2020-10-22T12:42:31.443 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.444 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.444 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", , 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.444 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", , 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.445 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "localhost", , 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.445 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.446 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.446 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", , 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.447 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, , 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", , 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.447 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { , 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", , 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2020-10-22T12:42:31.448 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", , 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.448 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.684 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", , 2020-10-22T12:24:55.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", , 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", , 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2020-10-22T12:42:31.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.726 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", , 2020-10-22T12:24:55.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.451 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: ], , 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: "services": {}, , 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: "always_on_modules": { 2020-10-22T12:42:31.452 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:25:08.182 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:25:08.183 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:25:08.183 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-22 12:24:22.295240 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.453 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:26:10.194 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:10.194 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:10.195 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.454 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:26:51.581 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2020-10-22 12:24:19.867927 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=19, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:51.581 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:27:11.582 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.456 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:27:31.719 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.f , instructing pg 1.f on osd.6 to deep-scrub , 2020-10-22T12:27:32.343 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.10 2020-10-22T12:42:31.457 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:28:35.645 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:28:35.645 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:28:35.646 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-10-22 12:24:20.879074 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.458 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:29:37.641 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:29:37.642 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:29:37.642 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.459 WARNING:scrape:Ignoring malparsed backtrace: instructing pg 1.11 on osd.0 to deep-scrub , 2020-10-22T12:29:58.966 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:30:18.968 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.461 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.461 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:32:02.940 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:02.940 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:02.941 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.463 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:32:43.699 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:43.700 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:43.700 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.464 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:33:25.035 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-22 12:24:29.921190 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:33:25.035 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:33:25.036 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-22 12:24:22.839282 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.465 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:34:26.998 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:34:26.999 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:34:26.999 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.466 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:35:08.808 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:08.808 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:08.809 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.467 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:35:50.124 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2020-10-22 12:24:19.867927 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=19, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:50.127 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:36:10.128 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.468 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:36:52.155 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:36:52.156 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:36:52.157 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.470 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:37:33.738 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:37:33.738 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:37:33.739 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-10-22 12:24:20.879074 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.471 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:38:35.692 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:38:35.693 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:38:35.693 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.472 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-22 12:24:29.921190 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-22 12:24:22.839282 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.473 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-22 12:24:22.295240 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.474 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:00.295 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:41:00.296 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:41:00.296 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.475 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:20.929 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-22T12:41:20.966 INFO:tasks.ceph.osd.0:Stopped , 2020-10-22T12:41:20.967 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-10-22T12:42:31.475 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:34.141 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1022/qa/tasks/ceph.py", line 1873, in task 2020-10-22T12:42:31.476 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:35.575 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-22T12:41:35.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com , 2020-10-22T12:41:35.577 INFO:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2020-10-22T12:42:31.506 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.11-85-g200fda8bb9 (200fda8bb95fcd626ae7aef62151966fb83c6c48) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7fa4543dc8d4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5be) [0x55af08a6c14e] 2020-10-22T12:42:31.507 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.318361", 2020-10-22T12:42:31.507 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { , 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", , 2020-10-22T12:24:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2020-10-22T12:42:31.508 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.324862", , 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg" , 2020-10-22T12:24:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.508 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4314.0:10 2.es0 2:798170f8:::smithi08938662-4:head [append 0~50421,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-10-22T12:42:31.508 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.160 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.327417", 2020-10-22T12:42:31.509 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] , 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:20.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.509 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.319697", , 2020-10-22T12:24:20.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg" , 2020-10-22T12:24:20.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.510 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done" , 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:20.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2020-10-22T12:42:31.510 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.368996", 2020-10-22T12:42:31.511 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.406670", , 2020-10-22T12:24:20.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_committed" 2020-10-22T12:42:31.512 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:20.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.512 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.029201202999999998, , 2020-10-22T12:24:20.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { , 2020-10-22T12:24:20.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2020-10-22T12:42:31.513 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.271 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.517485", , 2020-10-22T12:24:20.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg" 2020-10-22T12:42:31.513 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.548921", , 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done" , 2020-10-22T12:24:20.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2020-10-22T12:42:31.514 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.326558", , 2020-10-22T12:24:20.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched" , 2020-10-22T12:24:20.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.514 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.367087", , 2020-10-22T12:24:20.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched" 2020-10-22T12:42:31.515 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.378731", 2020-10-22T12:42:31.515 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" , 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.516 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.389624", , 2020-10-22T12:24:20.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" , 2020-10-22T12:24:20.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.516 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.473124", , 2020-10-22T12:24:20.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read" 2020-10-22T12:42:31.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:20.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.489919", 2020-10-22T12:42:31.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:21.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:21.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.535787", , 2020-10-22T12:24:21.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" , 2020-10-22T12:24:21.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.518 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: { , 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2020-10-22 12:24:19.536160", , 2020-10-22T12:24:21.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled" 2020-10-22T12:42:31.518 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.769 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "done" , 2020-10-22T12:24:21.770 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.770 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2020-10-22T12:42:31.519 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.785 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.313456", 2020-10-22T12:42:31.519 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.330772", , 2020-10-22T12:24:21.800 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" 2020-10-22T12:42:31.520 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.815 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.520 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.831 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "header_read" , 2020-10-22T12:24:21.831 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.832 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.521 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpWrite(2.ds1 25/22 ECSubWrite(tid=12, reqid=client.4314.0:49, at_version=25'4, trim_to=0'0, roll_forward_to=24'1))", , 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.340811", , 2020-10-22T12:24:21.847 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 2.4027936730000001, 2020-10-22T12:42:31.521 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpWrite(2.ds1 25/22 ECSubWrite(tid=20, reqid=client.4314.0:52, at_version=25'6, trim_to=0'0, roll_forward_to=25'3))", , 2020-10-22T12:24:21.862 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.344658", 2020-10-22T12:42:31.522 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.876 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.398794", , 2020-10-22T12:24:21.877 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_committed" , 2020-10-22T12:24:21.877 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.522 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.891 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "initiated" , 2020-10-22T12:24:21.891 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:21.892 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.523 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.906 INFO:teuthology.orchestra.run.smithi057.stdout: ] , 2020-10-22T12:24:21.906 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.523 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: } , 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: ] , 2020-10-22T12:24:21.921 INFO:teuthology.orchestra.run.smithi057.stdout: } 2020-10-22T12:42:31.524 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.508271", , 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:21.936 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.524 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.779 INFO:teuthology.orchestra.run.smithi057.stdout:{ , 2020-10-22T12:24:22.780 INFO:teuthology.orchestra.run.smithi057.stdout: "size": 20, , 2020-10-22T12:24:22.780 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 600, 2020-10-22T12:42:31.525 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.305767", , 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:22.795 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.525 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: "events": [ , 2020-10-22T12:24:22.811 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.526 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.826 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.394952", 2020-10-22T12:42:31.526 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.393377", , 2020-10-22T12:24:22.841 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" 2020-10-22T12:42:31.527 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "reached pg", , 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:22.857 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.527 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.873 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.406584", 2020-10-22T12:42:31.528 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.893 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.433674", , 2020-10-22T12:24:22.894 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" , 2020-10-22T12:24:22.894 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.528 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 3.3210408409999999, , 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 0.025615425000000001, , 2020-10-22T12:24:22.908 INFO:teuthology.orchestra.run.smithi057.stdout: "type_data": { 2020-10-22T12:42:31.528 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.923 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.438786", 2020-10-22T12:42:31.529 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: "client_addr": "172.21.15.89:0/1642317874", , 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: "tid": 117 , 2020-10-22T12:24:22.938 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.530 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:22.957 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:22.957 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:22.958 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "MOSDECSubOpReadReply(2.3s0 25/22 ECSubReadReply(tid=22, attrs_read=0))", 2020-10-22T12:42:31.530 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:23.937 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.311152", 2020-10-22T12:42:31.530 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:23.974 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:23.975 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:23.975 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.531 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.015 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.314272", , 2020-10-22T12:24:24.015 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" , 2020-10-22T12:24:24.016 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.531 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.324831", , 2020-10-22T12:24:24.037 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_started" 2020-10-22T12:42:31.532 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.061 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.342268", 2020-10-22T12:42:31.532 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "sub_op_committed" , 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.084 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.533 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.403405", , 2020-10-22T12:24:24.100 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "started" 2020-10-22T12:42:31.533 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "started", , 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:24.114 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.534 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.128 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.129 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.129 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.415018", 2020-10-22T12:42:31.534 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.446952", , 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:24.143 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.535 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 4.4306505979999997, , 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "duration": 0.036566672000000001, , 2020-10-22T12:24:24.159 INFO:teuthology.orchestra.run.smithi057.stdout: "type_data": { 2020-10-22T12:42:31.535 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.297809", 2020-10-22T12:42:31.536 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.324731", , 2020-10-22T12:24:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" 2020-10-22T12:42:31.536 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "flag_point": "started", , 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "client_info": { , 2020-10-22T12:24:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "client": "client.4314", 2020-10-22T12:42:31.537 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.537 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "description": "osd_op(client.4314.0:56 2.0s0 2:06f3166f:::smithi08938662-19 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [setxattr _header (357)] snapc 0=[] ondisk+write+known_if_redirected e25)", , 2020-10-22T12:24:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "initiated_at": "2020-10-22 12:24:19.337798", 2020-10-22T12:42:31.537 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.338020", , 2020-10-22T12:24:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "dispatched" 2020-10-22T12:42:31.538 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.389364", , 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "commit_sent" , 2020-10-22T12:24:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.538 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "dispatched" , 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.539 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.462113", 2020-10-22T12:42:31.539 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.426748", , 2020-10-22T12:24:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "all_read" 2020-10-22T12:42:31.540 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.463685", , 2020-10-22T12:24:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "reached_pg" , 2020-10-22T12:24:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.540 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "throttled" , 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: { 2020-10-22T12:42:31.541 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: }, , 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: { , 2020-10-22T12:24:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.497733", 2020-10-22T12:42:31.541 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "time": "2020-10-22 12:24:19.524406", , 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "event": "throttled" , 2020-10-22T12:24:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2020-10-22T12:42:31.543 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:50.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574865 got 55834574865 for osd.2 , 2020-10-22T12:24:50.119 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.3 , 2020-10-22T12:24:50.631 INFO:teuthology.orchestra.run.smithi089.stdout:55834574864 2020-10-22T12:42:31.543 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.047 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6789/0", , 2020-10-22T12:24:55.048 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6789/0" , 2020-10-22T12:24:55.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.544 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2020-10-22T12:42:31.544 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.075 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.545 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.090 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.090 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", , 2020-10-22T12:24:55.091 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-10-22T12:42:31.545 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] , 2020-10-22T12:24:55.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.545 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.121 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, , 2020-10-22T12:24:55.122 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2020-10-22T12:42:31.546 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_mirroring": { , 2020-10-22T12:24:55.135 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_mirroring", 2020-10-22T12:42:31.546 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.547 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.165 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.547 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.178 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.178 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", , 2020-10-22T12:24:55.179 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.548 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] , 2020-10-22T12:24:55.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.548 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", , 2020-10-22T12:24:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.549 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { , 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", , 2020-10-22T12:24:55.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2020-10-22T12:42:31.549 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.240 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.241 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.241 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.550 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.255 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.256 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2020-10-22T12:42:31.550 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.270 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.551 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { , 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", , 2020-10-22T12:24:55.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2020-10-22T12:42:31.551 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.300 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.552 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.314 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2020-10-22T12:42:31.552 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.328 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.329 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.329 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.553 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { , 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", , 2020-10-22T12:24:55.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2020-10-22T12:42:31.553 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.358 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.554 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.372 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2020-10-22T12:42:31.554 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.387 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.555 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", , 2020-10-22T12:24:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.555 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.556 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2020-10-22T12:42:31.556 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2020-10-22T12:42:31.556 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", , 2020-10-22T12:24:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", , 2020-10-22T12:24:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2020-10-22T12:42:31.557 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", , 2020-10-22T12:24:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.557 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", , 2020-10-22T12:24:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2020-10-22T12:42:31.558 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.558 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { , 2020-10-22T12:24:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2020-10-22T12:42:31.559 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.559 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", , 2020-10-22T12:24:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2020-10-22T12:42:31.560 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", , 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.560 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", , 2020-10-22T12:24:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.561 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "localhost", , 2020-10-22T12:24:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.561 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: } , 2020-10-22T12:24:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, , 2020-10-22T12:24:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2020-10-22T12:42:31.562 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", , 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], , 2020-10-22T12:24:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2020-10-22T12:42:31.562 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", , 2020-10-22T12:24:55.630 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.563 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, , 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", , 2020-10-22T12:24:55.644 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.563 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { , 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", , 2020-10-22T12:24:55.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2020-10-22T12:42:31.563 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", , 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.671 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.564 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.684 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], , 2020-10-22T12:24:55.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", , 2020-10-22T12:24:55.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2020-10-22T12:42:31.564 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", , 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", , 2020-10-22T12:24:55.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2020-10-22T12:42:31.565 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", , 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", , 2020-10-22T12:24:55.712 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2020-10-22T12:42:31.565 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, , 2020-10-22T12:24:55.726 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", , 2020-10-22T12:24:55.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2020-10-22T12:42:31.567 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: ], , 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: "services": {}, , 2020-10-22T12:24:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: "always_on_modules": { 2020-10-22T12:42:31.568 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:25:08.182 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:25:08.183 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:25:08.183 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-22 12:24:22.295240 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.569 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:26:10.194 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:10.194 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:10.195 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.570 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:26:51.581 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2020-10-22 12:24:19.867927 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=19, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:26:51.581 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:27:11.582 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.572 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:27:31.719 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.f , instructing pg 1.f on osd.6 to deep-scrub , 2020-10-22T12:27:32.343 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.10 2020-10-22T12:42:31.573 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:28:35.645 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:28:35.645 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:28:35.646 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-10-22 12:24:20.879074 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.574 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:29:37.641 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:29:37.642 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:29:37.642 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.575 WARNING:scrape:Ignoring malparsed backtrace: instructing pg 1.11 on osd.0 to deep-scrub , 2020-10-22T12:29:58.966 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:30:18.968 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.577 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:31:00.903 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.578 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:32:02.940 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:02.940 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:02.941 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.579 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:32:43.699 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:43.700 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:32:43.700 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.581 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:33:25.035 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-22 12:24:29.921190 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:33:25.035 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:33:25.036 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-22 12:24:22.839282 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.582 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:34:26.998 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:34:26.999 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:34:26.999 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.583 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:35:08.808 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-22 12:24:24.082364 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=24, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:08.808 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:08.809 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.584 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:35:50.124 INFO:tasks.ceph:pgid 1.11 last_scrub_stamp 2020-10-22 12:24:19.867927 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=19, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:35:50.127 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-22T12:36:10.128 INFO:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2020-10-22T12:42:31.585 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:36:52.155 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:36:52.156 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-22 12:24:29.681478 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:36:52.157 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.587 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:37:33.738 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-22 12:24:27.633002 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=27, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:37:33.738 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:37:33.739 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-10-22 12:24:20.879074 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.588 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:38:35.692 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-22 12:24:26.858037 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=26, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:38:35.693 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:38:35.693 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-22 12:24:40.614465 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=40, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.589 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-22 12:24:29.921190 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=29, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:16.996 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-22 12:24:22.839282 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.590 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-22 12:24:22.295240 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=22, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:39:58.275 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.591 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:00.295 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:41:00.296 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-22 12:24:20.827548 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=20, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) , 2020-10-22T12:41:00.296 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-22 12:24:16.231488 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=16, tm_wday=3, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=22, tm_hour=12, tm_min=24, tm_sec=58, tm_wday=3, tm_yday=296, tm_isdst=0) 2020-10-22T12:42:31.592 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:20.929 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-22T12:41:20.966 INFO:tasks.ceph.osd.0:Stopped , 2020-10-22T12:41:20.967 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-10-22T12:42:31.592 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:34.141 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1022/qa/tasks/ceph.py", line 1873, in task 2020-10-22T12:42:31.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-22T12:41:35.575 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-22T12:41:35.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com , 2020-10-22T12:41:35.577 INFO:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2020-10-22T12:42:31.607 INFO:scrape:Found 1 distinct failure reasons 2020-10-22T12:42:31.607 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2020-10-22T12:42:31.608 INFO:scrape:1 jobs: ['5547961'] 2020-10-22T12:42:31.608 INFO:scrape:suites: ['clusters/{fixed-2.yaml', 'fast/fast.yaml', 'msgr-failures/osd-delay.yaml', 'objectstore/bluestore-comp-zlib.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash-erasure-code/{ceph.yaml', 'recovery-overrides/{more-partial-recovery.yaml}', 'supported-random-distro$/{centos_latest.yaml}', 'thrashers/morepggrow.yaml', 'thrashosds-health.yaml', 'workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml}'] 2020-10-22T12:42:31.608 INFO:scrape: