2020-07-15T05:25:17.462 INFO:root:teuthology version: 1.0.0-da68153 2020-07-15T05:25:17.464 INFO:teuthology.results:subset: '523/1635' 2020-07-15T05:25:17.464 INFO:teuthology.results:seed: '4993' 2020-07-15T07:15:39.031 INFO:root:teuthology version: 1.0.0-da68153 2020-07-15T07:15:39.050 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-07-15T08:56:42.988 INFO:teuthology.results:Tests finished! gathering results... 2020-07-15T08:57:05.086 INFO:scrape:Found 249 jobs 2020-07-15T08:57:05.253 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.10-197-g6feeb409f65 (6feeb409f65c62db8056e0a029baa53459633c2c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f95dbe239bc] , 2: (CopyOp::_finish(TestOp::CallbackInfo*)+0x1c1) [0x5639533f5c51] 2020-07-15T08:57:05.254 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.629 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:55.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.257 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: ] , 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.258 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" , 2020-07-15T06:59:55.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.259 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.710 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.635255", 2020-07-15T08:57:05.260 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.735 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.635327", , 2020-07-15T06:59:55.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" 2020-07-15T08:57:05.260 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.637162", , 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated" , 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.261 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.9665948200000001, , 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.18281774200000001, , 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2020-07-15T08:57:05.262 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.820451", 2020-07-15T08:57:05.264 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.639782", , 2020-07-15T06:59:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" 2020-07-15T08:57:05.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.640677", , 2020-07-15T06:59:55.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" , 2020-07-15T06:59:55.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated" , 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.221 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1335 2.2 2:479cc21c:::smithi19040819-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:56.222 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:23.956900", , 2020-07-15T06:59:56.222 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.266734671000002, 2020-07-15T08:57:05.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:23.964116", , 2020-07-15T06:59:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 2" 2020-07-15T08:57:05.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1338 , 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2020-07-15T08:57:05.268 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started" , 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.269 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4492", , 2020-07-15T06:59:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "v1:172.21.15.190:0/391360727", , 2020-07-15T06:59:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1348 2020-07-15T08:57:05.269 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.035162", , 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started" 2020-07-15T08:57:05.270 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-07-15T06:59:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { , 2020-07-15T06:59:56.510 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4492", 2020-07-15T08:57:05.271 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.535 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" , 2020-07-15T06:59:56.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.536 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.271 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.020697970999999999, , 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { , 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2020-07-15T08:57:05.272 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.594 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.162838", , 2020-07-15T06:59:56.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" 2020-07-15T08:57:05.272 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.616 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:24.229876", , 2020-07-15T06:59:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.157471346999998, , 2020-07-15T06:59:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022189284, 2020-07-15T08:57:05.273 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" , 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.274 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.665 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.666 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1398 2.3 2:daee9c7f:::smithi19040819-67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 13] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:56.666 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:24.332698", 2020-07-15T08:57:05.274 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.694 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.442704", , 2020-07-15T06:59:56.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" 2020-07-15T08:57:05.275 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.726 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:56.727 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:56.727 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-07-15T08:57:05.276 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1343 2.0 2:2fb592ca:::smithi19040819-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:23.993592", , 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.29132079, 2020-07-15T08:57:05.276 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.005208", , 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" , 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.277 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1363 2.0 2:3b8cfef3:::smithi19040819-32 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-07-15T08:57:05.277 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.144738", 2020-07-15T08:57:05.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: ] , 2020-07-15T06:59:57.409 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-07-15T08:57:05.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.213217", , 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched" 2020-07-15T08:57:05.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.234016", , 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done" , 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-07-15T08:57:05.280 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" , 2020-07-15T06:59:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.494 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.494 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.385551", 2020-07-15T08:57:05.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.512 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.513 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.402632", , 2020-07-15T06:59:57.513 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" 2020-07-15T08:57:05.282 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.531 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.429846", , 2020-07-15T06:59:57.532 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent" , 2020-07-15T06:59:57.532 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.283 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.550 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled" , 2020-07-15T06:59:57.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.283 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.569 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.570 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.495794", 2020-07-15T08:57:05.286 WARNING:scrape:Ignoring malparsed backtrace: mon.c: injectargs:mon_osd_down_out_interval = '600' (not observed, change may require restart) , 2020-07-15T07:00:04.639 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 0 1 , reweighted osd.0 to 1 (10000) 2020-07-15T08:57:05.286 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.881 INFO:teuthology.orchestra.run.smithi190.stdout: "severity": "HEALTH_WARN", , 2020-07-15T07:00:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: "summary": { , 2020-07-15T07:00:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: "message": "application not enabled on 1 pool(s)" 2020-07-15T08:57:05.287 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.909 INFO:teuthology.orchestra.run.smithi190.stdout: "state_name": "active+clean+scrubbing+deep", , 2020-07-15T07:00:28.909 INFO:teuthology.orchestra.run.smithi190.stdout: "count": 1 , 2020-07-15T07:00:28.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-07-15T08:57:05.288 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 1, 2020-07-15T08:57:05.288 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: "username": { 2020-07-15T08:57:05.289 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "metrics with which to calculate OSD utilization", , 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2020-07-15T08:57:05.290 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "how frequently to wake up and attempt optimization", 2020-07-15T08:57:05.291 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.292 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { , 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2020-07-15T08:57:05.292 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.293 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.191 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "", , 2020-07-15T07:00:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.294 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.294 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.276 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.277 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.277 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.295 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.318 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.318 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.295 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: } , 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-07-15T08:57:05.296 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.297 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": { , 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "batch_size": { , 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "batch_size", 2020-07-15T08:57:05.297 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.399 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:29.400 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.400 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.298 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "pg_num": { , 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "pg_num", , 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "int", 2020-07-15T08:57:05.298 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} , 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-07-15T08:57:05.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "testnewline": { , 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "testnewline", , 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", 2020-07-15T08:57:05.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, , 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "True", , 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", 2020-07-15T08:57:05.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.595 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.595 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.596 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", 2020-07-15T08:57:05.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, , 2020-07-15T07:00:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: "error_string": "", , 2020-07-15T07:00:29.635 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} 2020-07-15T08:57:05.304 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.665 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "How frequently to reexamine health status", , 2020-07-15T07:00:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.687 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.687 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "begin_time": { , 2020-07-15T07:00:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "begin_time", , 2020-07-15T07:00:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", 2020-07-15T08:57:05.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 1, , 2020-07-15T07:00:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "0", 2020-07-15T08:57:05.307 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "how long to retain crashes before pruning them", , 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], 2020-07-15T08:57:05.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "True", , 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "debug": { , 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "debug", , 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "bool", 2020-07-15T08:57:05.310 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.939 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.940 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, , 2020-07-15T07:00:29.940 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "8443", 2020-07-15T08:57:05.311 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], 2020-07-15T08:57:05.313 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "600", , 2020-07-15T07:00:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.315 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "error_string": "", , 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} , 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.315 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.316 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "scrape_interval": { 2020-07-15T08:57:05.317 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: { , 2020-07-15T07:00:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "rook", , 2020-07-15T07:00:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, 2020-07-15T08:57:05.317 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.158 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:30.158 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:30.159 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], 2020-07-15T08:57:05.318 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.179 INFO:teuthology.orchestra.run.smithi190.stdout: { , 2020-07-15T07:00:30.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "status", , 2020-07-15T07:00:30.180 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, 2020-07-15T08:57:05.319 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:30.199 INFO:teuthology.orchestra.run.smithi190.stdout: "contact": { , 2020-07-15T07:00:30.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "contact", 2020-07-15T08:57:05.319 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.217 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.218 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.218 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "zabbix_port": { , 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "zabbix_port", , 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "int", 2020-07-15T08:57:05.324 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:36.760 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 , 2020-07-15T07:00:36.989 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 6 2020-07-15T08:57:05.325 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:39.769 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell mon.* injectargs -- --no-mon-health-to-clog , mon.b: injectargs:mon_health_to_clog = 'false' , mon.a: injectargs:mon_health_to_clog = 'false' 2020-07-15T08:57:05.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:43.502 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.MCEMJwP5vA , 2020-07-15T07:01:43.503 INFO:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MCEMJwP5vA , 2020-07-15T07:01:43.671 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.MCEMJwP5vA 2020-07-15T08:57:05.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary , 2020-07-15T07:01:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ , 2020-07-15T07:01:51.669 INFO:teuthology.orchestra.run.smithi190.stdout:Remove 1 Package 2020-07-15T08:57:05.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:56.277 INFO:teuthology.orchestra.run.smithi164.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- , 2020-07-15T07:01:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: : manager , 2020-07-15T07:01:56.343 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:ceph-test-14.2.10-197.g6feeb409f65.el7.x86_64 1/1 2020-07-15T08:57:05.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:02.358 INFO:teuthology.orchestra.run.smithi164.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-mgr-diskprediction-cloud-14.2.10-197.g6feeb409f65.el7.noarch , 2020-07-15T07:02:02.361 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-mgr-rook-14.2.10-197.g6feeb409f65.el7.noarch , 2020-07-15T07:02:02.377 INFO:teuthology.orchestra.run.smithi164.stdout:--> Running transaction check 2020-07-15T08:57:05.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout: , 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout:Dependency Removed: 2020-07-15T08:57:05.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ , 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout:Removing: , 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:14.2.10-197.g6feeb409f65.el7 @ceph 1.6 M 2020-07-15T08:57:05.330 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:16.167 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-base-14.2.10-197.g6feeb409f65.el7.x86_64 , 2020-07-15T07:02:16.167 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-common-14.2.10-197.g6feeb409f65.el7.x86_64 , 2020-07-15T07:02:16.186 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:python-cephfs-14.2.10-197.g6feeb409f65.el7.x86_64 2020-07-15T08:57:05.330 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: python-ceph-compat.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: python-cephfs.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-07-15T08:57:05.331 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: qemu-kvm x86_64 10:1.5.3-173.el7_8.3 @rhel-7-server-rpms 5.7 M , 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: qemu-kvm-common , 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 10:1.5.3-173.el7_8.3 @rhel-7-server-rpms 755 k 2020-07-15T08:57:05.332 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:32.254 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:python-rgw-14.2.10-197.g6feeb409f65.el7.x86_64 2/12 , 2020-07-15T07:02:32.523 INFO:teuthology.orchestra.run.smithi164.stdout: Erasing : 2:python-rados-14.2.10-197.g6feeb409f65.el7.x86_64 8/12 , 2020-07-15T07:02:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:python-rbd-14.2.10-197.g6feeb409f65.el7.x86_64 3/12 2020-07-15T08:57:05.332 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:38.094 INFO:teuthology.orchestra.run.smithi190.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- , 2020-07-15T07:02:38.095 INFO:teuthology.orchestra.run.smithi190.stdout: : disabled-repos, subscription-manager , 2020-07-15T07:02:38.386 INFO:teuthology.orchestra.run.smithi164.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2020-07-15T08:57:05.333 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:42.013 INFO:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2020-07-15T07:02:42.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2020-07-15T07:02:42.102 INFO:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2020-07-15T08:57:05.407 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.10-197-g6feeb409f65 (6feeb409f65c62db8056e0a029baa53459633c2c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f95dbe239bc] , 2: (CopyOp::_finish(TestOp::CallbackInfo*)+0x1c1) [0x5639533f5c51] 2020-07-15T08:57:05.410 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.629 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:55.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.411 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: ] , 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.412 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" , 2020-07-15T06:59:55.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.412 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.710 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.635255", 2020-07-15T08:57:05.413 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.735 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.635327", , 2020-07-15T06:59:55.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" 2020-07-15T08:57:05.414 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.637162", , 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated" , 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.414 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.9665948200000001, , 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.18281774200000001, , 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2020-07-15T08:57:05.415 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.820451", 2020-07-15T08:57:05.416 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.639782", , 2020-07-15T06:59:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" 2020-07-15T08:57:05.417 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.640677", , 2020-07-15T06:59:55.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" , 2020-07-15T06:59:55.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.418 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated" , 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.418 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.221 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1335 2.2 2:479cc21c:::smithi19040819-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:56.222 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:23.956900", , 2020-07-15T06:59:56.222 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.266734671000002, 2020-07-15T08:57:05.419 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:23.964116", , 2020-07-15T06:59:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 2" 2020-07-15T08:57:05.420 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1338 , 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2020-07-15T08:57:05.421 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started" , 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.421 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4492", , 2020-07-15T06:59:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "v1:172.21.15.190:0/391360727", , 2020-07-15T06:59:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1348 2020-07-15T08:57:05.422 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.035162", , 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started" 2020-07-15T08:57:05.423 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-07-15T06:59:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { , 2020-07-15T06:59:56.510 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4492", 2020-07-15T08:57:05.424 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.535 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" , 2020-07-15T06:59:56.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.536 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.424 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.020697970999999999, , 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { , 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2020-07-15T08:57:05.425 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.594 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.162838", , 2020-07-15T06:59:56.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" 2020-07-15T08:57:05.426 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.616 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:24.229876", , 2020-07-15T06:59:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.157471346999998, , 2020-07-15T06:59:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022189284, 2020-07-15T08:57:05.428 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" , 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.429 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.665 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.666 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1398 2.3 2:daee9c7f:::smithi19040819-67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 13] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:56.666 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:24.332698", 2020-07-15T08:57:05.430 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.694 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.442704", , 2020-07-15T06:59:56.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" 2020-07-15T08:57:05.431 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.726 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:56.727 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:56.727 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-07-15T08:57:05.432 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1343 2.0 2:2fb592ca:::smithi19040819-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:23.993592", , 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.29132079, 2020-07-15T08:57:05.433 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.005208", , 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" , 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.434 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1363 2.0 2:3b8cfef3:::smithi19040819-32 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-07-15T08:57:05.435 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.144738", 2020-07-15T08:57:05.436 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: ] , 2020-07-15T06:59:57.409 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-07-15T08:57:05.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.213217", , 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched" 2020-07-15T08:57:05.438 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.234016", , 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done" , 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-07-15T08:57:05.439 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" , 2020-07-15T06:59:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.494 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.494 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.385551", 2020-07-15T08:57:05.441 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.512 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.513 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.402632", , 2020-07-15T06:59:57.513 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" 2020-07-15T08:57:05.442 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.531 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.429846", , 2020-07-15T06:59:57.532 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent" , 2020-07-15T06:59:57.532 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.443 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.550 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled" , 2020-07-15T06:59:57.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.444 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.569 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.570 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.495794", 2020-07-15T08:57:05.447 WARNING:scrape:Ignoring malparsed backtrace: mon.c: injectargs:mon_osd_down_out_interval = '600' (not observed, change may require restart) , 2020-07-15T07:00:04.639 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 0 1 , reweighted osd.0 to 1 (10000) 2020-07-15T08:57:05.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.881 INFO:teuthology.orchestra.run.smithi190.stdout: "severity": "HEALTH_WARN", , 2020-07-15T07:00:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: "summary": { , 2020-07-15T07:00:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: "message": "application not enabled on 1 pool(s)" 2020-07-15T08:57:05.451 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.909 INFO:teuthology.orchestra.run.smithi190.stdout: "state_name": "active+clean+scrubbing+deep", , 2020-07-15T07:00:28.909 INFO:teuthology.orchestra.run.smithi190.stdout: "count": 1 , 2020-07-15T07:00:28.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-07-15T08:57:05.452 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 1, 2020-07-15T08:57:05.453 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: "username": { 2020-07-15T08:57:05.455 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "metrics with which to calculate OSD utilization", , 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2020-07-15T08:57:05.456 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "how frequently to wake up and attempt optimization", 2020-07-15T08:57:05.457 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.458 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { , 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2020-07-15T08:57:05.458 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.459 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.191 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "", , 2020-07-15T07:00:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.460 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.461 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.276 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.277 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.277 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.462 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.318 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.318 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.462 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: } , 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-07-15T08:57:05.463 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.463 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": { , 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "batch_size": { , 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "batch_size", 2020-07-15T08:57:05.464 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.399 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:29.400 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.400 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.465 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "pg_num": { , 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "pg_num", , 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "int", 2020-07-15T08:57:05.465 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.466 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} , 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-07-15T08:57:05.467 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.468 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "testnewline": { , 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "testnewline", , 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", 2020-07-15T08:57:05.469 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, , 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "True", , 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", 2020-07-15T08:57:05.470 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.595 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.595 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.596 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", 2020-07-15T08:57:05.470 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, , 2020-07-15T07:00:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: "error_string": "", , 2020-07-15T07:00:29.635 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} 2020-07-15T08:57:05.471 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.665 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "How frequently to reexamine health status", , 2020-07-15T07:00:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.472 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.687 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.687 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.473 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "begin_time": { , 2020-07-15T07:00:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "begin_time", , 2020-07-15T07:00:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", 2020-07-15T08:57:05.474 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 1, , 2020-07-15T07:00:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "0", 2020-07-15T08:57:05.475 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "how long to retain crashes before pruning them", , 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], 2020-07-15T08:57:05.477 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "True", , 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.479 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.480 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.481 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "debug": { , 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "debug", , 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "bool", 2020-07-15T08:57:05.481 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.939 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.940 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, , 2020-07-15T07:00:29.940 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "8443", 2020-07-15T08:57:05.482 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.483 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], 2020-07-15T08:57:05.484 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.487 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "600", , 2020-07-15T07:00:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.489 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.490 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "error_string": "", , 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} , 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.491 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.492 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "scrape_interval": { 2020-07-15T08:57:05.492 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: { , 2020-07-15T07:00:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "rook", , 2020-07-15T07:00:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, 2020-07-15T08:57:05.493 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.158 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:30.158 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:30.159 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], 2020-07-15T08:57:05.494 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.179 INFO:teuthology.orchestra.run.smithi190.stdout: { , 2020-07-15T07:00:30.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "status", , 2020-07-15T07:00:30.180 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, 2020-07-15T08:57:05.495 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:30.199 INFO:teuthology.orchestra.run.smithi190.stdout: "contact": { , 2020-07-15T07:00:30.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "contact", 2020-07-15T08:57:05.495 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.217 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.218 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.218 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.497 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "zabbix_port": { , 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "zabbix_port", , 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "int", 2020-07-15T08:57:05.501 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:36.760 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 , 2020-07-15T07:00:36.989 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 6 2020-07-15T08:57:05.502 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:39.769 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell mon.* injectargs -- --no-mon-health-to-clog , mon.b: injectargs:mon_health_to_clog = 'false' , mon.a: injectargs:mon_health_to_clog = 'false' 2020-07-15T08:57:05.502 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:43.502 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.MCEMJwP5vA , 2020-07-15T07:01:43.503 INFO:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MCEMJwP5vA , 2020-07-15T07:01:43.671 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.MCEMJwP5vA 2020-07-15T08:57:05.503 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary , 2020-07-15T07:01:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ , 2020-07-15T07:01:51.669 INFO:teuthology.orchestra.run.smithi190.stdout:Remove 1 Package 2020-07-15T08:57:05.504 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:56.277 INFO:teuthology.orchestra.run.smithi164.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- , 2020-07-15T07:01:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: : manager , 2020-07-15T07:01:56.343 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:ceph-test-14.2.10-197.g6feeb409f65.el7.x86_64 1/1 2020-07-15T08:57:05.505 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:02.358 INFO:teuthology.orchestra.run.smithi164.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-mgr-diskprediction-cloud-14.2.10-197.g6feeb409f65.el7.noarch , 2020-07-15T07:02:02.361 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-mgr-rook-14.2.10-197.g6feeb409f65.el7.noarch , 2020-07-15T07:02:02.377 INFO:teuthology.orchestra.run.smithi164.stdout:--> Running transaction check 2020-07-15T08:57:05.506 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout: , 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout:Dependency Removed: 2020-07-15T08:57:05.506 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ , 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout:Removing: , 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:14.2.10-197.g6feeb409f65.el7 @ceph 1.6 M 2020-07-15T08:57:05.507 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:16.167 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-base-14.2.10-197.g6feeb409f65.el7.x86_64 , 2020-07-15T07:02:16.167 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-common-14.2.10-197.g6feeb409f65.el7.x86_64 , 2020-07-15T07:02:16.186 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:python-cephfs-14.2.10-197.g6feeb409f65.el7.x86_64 2020-07-15T08:57:05.508 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: python-ceph-compat.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: python-cephfs.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-07-15T08:57:05.508 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: qemu-kvm x86_64 10:1.5.3-173.el7_8.3 @rhel-7-server-rpms 5.7 M , 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: qemu-kvm-common , 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 10:1.5.3-173.el7_8.3 @rhel-7-server-rpms 755 k 2020-07-15T08:57:05.509 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:32.254 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:python-rgw-14.2.10-197.g6feeb409f65.el7.x86_64 2/12 , 2020-07-15T07:02:32.523 INFO:teuthology.orchestra.run.smithi164.stdout: Erasing : 2:python-rados-14.2.10-197.g6feeb409f65.el7.x86_64 8/12 , 2020-07-15T07:02:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:python-rbd-14.2.10-197.g6feeb409f65.el7.x86_64 3/12 2020-07-15T08:57:05.510 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:38.094 INFO:teuthology.orchestra.run.smithi190.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- , 2020-07-15T07:02:38.095 INFO:teuthology.orchestra.run.smithi190.stdout: : disabled-repos, subscription-manager , 2020-07-15T07:02:38.386 INFO:teuthology.orchestra.run.smithi164.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2020-07-15T08:57:05.511 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:42.013 INFO:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2020-07-15T07:02:42.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2020-07-15T07:02:42.102 INFO:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2020-07-15T08:57:05.618 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.10-197-g6feeb409f65 (6feeb409f65c62db8056e0a029baa53459633c2c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f95dbe239bc] , 2: (CopyOp::_finish(TestOp::CallbackInfo*)+0x1c1) [0x5639533f5c51] 2020-07-15T08:57:05.620 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.629 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:55.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: ] , 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:55.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" , 2020-07-15T06:59:55.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.710 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.635255", 2020-07-15T08:57:05.624 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.735 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.635327", , 2020-07-15T06:59:55.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" 2020-07-15T08:57:05.624 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.637162", , 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated" , 2020-07-15T06:59:55.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.625 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.9665948200000001, , 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.18281774200000001, , 2020-07-15T06:59:55.793 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2020-07-15T08:57:05.626 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.820451", 2020-07-15T08:57:05.627 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:55.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.639782", , 2020-07-15T06:59:55.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" 2020-07-15T08:57:05.629 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:52.640677", , 2020-07-15T06:59:55.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" , 2020-07-15T06:59:55.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.630 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated" , 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:55.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.631 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.221 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1335 2.2 2:479cc21c:::smithi19040819-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:56.222 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:23.956900", , 2020-07-15T06:59:56.222 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.266734671000002, 2020-07-15T08:57:05.632 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:23.964116", , 2020-07-15T06:59:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 2" 2020-07-15T08:57:05.633 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1338 , 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2020-07-15T08:57:05.634 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started" , 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.635 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4492", , 2020-07-15T06:59:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "v1:172.21.15.190:0/391360727", , 2020-07-15T06:59:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1348 2020-07-15T08:57:05.636 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.035162", , 2020-07-15T06:59:56.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started" 2020-07-15T08:57:05.637 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-07-15T06:59:56.509 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { , 2020-07-15T06:59:56.510 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.4492", 2020-07-15T08:57:05.638 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.535 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" , 2020-07-15T06:59:56.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.536 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.639 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.020697970999999999, , 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { , 2020-07-15T06:59:56.573 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2020-07-15T08:57:05.640 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.594 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.162838", , 2020-07-15T06:59:56.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg" 2020-07-15T08:57:05.641 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.616 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:24.229876", , 2020-07-15T06:59:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.157471346999998, , 2020-07-15T06:59:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022189284, 2020-07-15T08:57:05.642 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" , 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:56.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.643 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.665 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.666 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1398 2.3 2:daee9c7f:::smithi19040819-67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 13] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:56.666 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:24.332698", 2020-07-15T08:57:05.644 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.694 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:56.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.442704", , 2020-07-15T06:59:56.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" 2020-07-15T08:57:05.645 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:56.726 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:56.727 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:56.727 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2020-07-15T08:57:05.646 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1343 2.0 2:2fb592ca:::smithi19040819-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected e26)", , 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2020-07-15 06:59:23.993592", , 2020-07-15T06:59:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 33.29132079, 2020-07-15T08:57:05.647 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.005208", , 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg" , 2020-07-15T06:59:57.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.647 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.4492.0:1363 2.0 2:3b8cfef3:::smithi19040819-32 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-07-15T08:57:05.648 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.144738", 2020-07-15T08:57:05.649 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: } , 2020-07-15T06:59:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: ] , 2020-07-15T06:59:57.409 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-07-15T08:57:05.649 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.213217", , 2020-07-15T06:59:57.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched" 2020-07-15T08:57:05.650 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.234016", , 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done" , 2020-07-15T06:59:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-07-15T08:57:05.651 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" , 2020-07-15T06:59:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.651 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.494 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.494 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.385551", 2020-07-15T08:57:05.652 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.512 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.513 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.402632", , 2020-07-15T06:59:57.513 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read" 2020-07-15T08:57:05.652 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.531 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.429846", , 2020-07-15T06:59:57.532 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent" , 2020-07-15T06:59:57.532 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-07-15T08:57:05.653 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.550 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled" , 2020-07-15T06:59:57.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2020-07-15T08:57:05.654 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T06:59:57.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, , 2020-07-15T06:59:57.569 INFO:teuthology.orchestra.run.smithi164.stdout: { , 2020-07-15T06:59:57.570 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2020-07-15 06:59:24.495794", 2020-07-15T08:57:05.656 WARNING:scrape:Ignoring malparsed backtrace: mon.c: injectargs:mon_osd_down_out_interval = '600' (not observed, change may require restart) , 2020-07-15T07:00:04.639 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 0 1 , reweighted osd.0 to 1 (10000) 2020-07-15T08:57:05.657 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.881 INFO:teuthology.orchestra.run.smithi190.stdout: "severity": "HEALTH_WARN", , 2020-07-15T07:00:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: "summary": { , 2020-07-15T07:00:28.882 INFO:teuthology.orchestra.run.smithi190.stdout: "message": "application not enabled on 1 pool(s)" 2020-07-15T08:57:05.657 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.909 INFO:teuthology.orchestra.run.smithi190.stdout: "state_name": "active+clean+scrubbing+deep", , 2020-07-15T07:00:28.909 INFO:teuthology.orchestra.run.smithi190.stdout: "count": 1 , 2020-07-15T07:00:28.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2020-07-15T08:57:05.658 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 1, 2020-07-15T08:57:05.658 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:28.967 INFO:teuthology.orchestra.run.smithi190.stdout: "username": { 2020-07-15T08:57:05.659 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "metrics with which to calculate OSD utilization", , 2020-07-15T07:00:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2020-07-15T08:57:05.660 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "how frequently to wake up and attempt optimization", 2020-07-15T08:57:05.660 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.055 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.661 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { , 2020-07-15T07:00:29.100 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2020-07-15T08:57:05.661 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.145 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.662 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.191 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "", , 2020-07-15T07:00:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.663 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.233 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.663 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.276 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.277 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.277 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.665 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.318 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.318 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.665 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: } , 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-07-15T08:57:05.666 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.361 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.666 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": { , 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "batch_size": { , 2020-07-15T07:00:29.381 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "batch_size", 2020-07-15T08:57:05.667 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.399 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:29.400 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.400 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.668 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "pg_num": { , 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "pg_num", , 2020-07-15T07:00:29.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "int", 2020-07-15T08:57:05.668 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.451 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.669 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} , 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:29.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2020-07-15T08:57:05.669 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.670 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "testnewline": { , 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "testnewline", , 2020-07-15T07:00:29.534 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", 2020-07-15T08:57:05.671 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, , 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "True", , 2020-07-15T07:00:29.562 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", 2020-07-15T08:57:05.671 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.595 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.595 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.596 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", 2020-07-15T08:57:05.672 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, , 2020-07-15T07:00:29.634 INFO:teuthology.orchestra.run.smithi190.stdout: "error_string": "", , 2020-07-15T07:00:29.635 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} 2020-07-15T08:57:05.672 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.665 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "How frequently to reexamine health status", , 2020-07-15T07:00:29.666 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.673 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.687 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.687 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.674 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "begin_time": { , 2020-07-15T07:00:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "begin_time", , 2020-07-15T07:00:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", 2020-07-15T08:57:05.674 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 1, , 2020-07-15T07:00:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "0", 2020-07-15T08:57:05.675 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "how long to retain crashes before pruning them", , 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], 2020-07-15T08:57:05.675 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "True", , 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.838 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.676 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:29.865 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.676 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:29.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.677 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "debug": { , 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "debug", , 2020-07-15T07:00:29.920 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "bool", 2020-07-15T08:57:05.678 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.939 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:29.940 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, , 2020-07-15T07:00:29.940 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "8443", 2020-07-15T08:57:05.678 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", , 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "tags": [], , 2020-07-15T07:00:29.962 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] 2020-07-15T08:57:05.679 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], 2020-07-15T08:57:05.679 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.680 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "default_value": "600", , 2020-07-15T07:00:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", 2020-07-15T08:57:05.681 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], , 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "desc": "", , 2020-07-15T07:00:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "long_desc": "", 2020-07-15T08:57:05.681 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "error_string": "", , 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "module_options": {} , 2020-07-15T07:00:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2020-07-15T08:57:05.682 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.682 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "see_also": [] , 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "scrape_interval": { 2020-07-15T08:57:05.683 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: { , 2020-07-15T07:00:30.138 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "rook", , 2020-07-15T07:00:30.139 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, 2020-07-15T08:57:05.684 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.158 INFO:teuthology.orchestra.run.smithi190.stdout: "min": "", , 2020-07-15T07:00:30.158 INFO:teuthology.orchestra.run.smithi190.stdout: "max": "", , 2020-07-15T07:00:30.159 INFO:teuthology.orchestra.run.smithi190.stdout: "enum_allowed": [], 2020-07-15T08:57:05.684 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.179 INFO:teuthology.orchestra.run.smithi190.stdout: { , 2020-07-15T07:00:30.180 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "status", , 2020-07-15T07:00:30.180 INFO:teuthology.orchestra.run.smithi190.stdout: "can_run": true, 2020-07-15T08:57:05.685 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, , 2020-07-15T07:00:30.199 INFO:teuthology.orchestra.run.smithi190.stdout: "contact": { , 2020-07-15T07:00:30.199 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "contact", 2020-07-15T08:57:05.686 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.217 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "str", , 2020-07-15T07:00:30.218 INFO:teuthology.orchestra.run.smithi190.stdout: "level": "advanced", , 2020-07-15T07:00:30.218 INFO:teuthology.orchestra.run.smithi190.stdout: "flags": 0, 2020-07-15T08:57:05.687 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "zabbix_port": { , 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "name": "zabbix_port", , 2020-07-15T07:00:30.236 INFO:teuthology.orchestra.run.smithi190.stdout: "type": "int", 2020-07-15T08:57:05.690 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:00:36.760 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 , 2020-07-15T07:00:36.989 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 6 2020-07-15T08:57:05.691 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:39.769 INFO:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell mon.* injectargs -- --no-mon-health-to-clog , mon.b: injectargs:mon_health_to_clog = 'false' , mon.a: injectargs:mon_health_to_clog = 'false' 2020-07-15T08:57:05.692 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:43.502 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.MCEMJwP5vA , 2020-07-15T07:01:43.503 INFO:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MCEMJwP5vA , 2020-07-15T07:01:43.671 INFO:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.MCEMJwP5vA 2020-07-15T08:57:05.692 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary , 2020-07-15T07:01:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ , 2020-07-15T07:01:51.669 INFO:teuthology.orchestra.run.smithi190.stdout:Remove 1 Package 2020-07-15T08:57:05.693 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:01:56.277 INFO:teuthology.orchestra.run.smithi164.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- , 2020-07-15T07:01:56.277 INFO:teuthology.orchestra.run.smithi164.stdout: : manager , 2020-07-15T07:01:56.343 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:ceph-test-14.2.10-197.g6feeb409f65.el7.x86_64 1/1 2020-07-15T08:57:05.694 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:02.358 INFO:teuthology.orchestra.run.smithi164.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-mgr-diskprediction-cloud-14.2.10-197.g6feeb409f65.el7.noarch , 2020-07-15T07:02:02.361 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-mgr-rook-14.2.10-197.g6feeb409f65.el7.noarch , 2020-07-15T07:02:02.377 INFO:teuthology.orchestra.run.smithi164.stdout:--> Running transaction check 2020-07-15T08:57:05.694 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout: , 2020-07-15T07:02:07.274 INFO:teuthology.orchestra.run.smithi164.stdout:Dependency Removed: 2020-07-15T08:57:05.695 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ , 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout:Removing: , 2020-07-15T07:02:12.599 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:14.2.10-197.g6feeb409f65.el7 @ceph 1.6 M 2020-07-15T08:57:05.696 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:16.167 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-base-14.2.10-197.g6feeb409f65.el7.x86_64 , 2020-07-15T07:02:16.167 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:ceph-common-14.2.10-197.g6feeb409f65.el7.x86_64 , 2020-07-15T07:02:16.186 INFO:teuthology.orchestra.run.smithi190.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.10-197.g6feeb409f65.el7 for package: 2:python-cephfs-14.2.10-197.g6feeb409f65.el7.x86_64 2020-07-15T08:57:05.696 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: python-ceph-compat.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: python-cephfs.x86_64 2:14.2.10-197.g6feeb409f65.el7 , 2020-07-15T07:02:27.705 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-07-15T08:57:05.697 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: qemu-kvm x86_64 10:1.5.3-173.el7_8.3 @rhel-7-server-rpms 5.7 M , 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: qemu-kvm-common , 2020-07-15T07:02:29.974 INFO:teuthology.orchestra.run.smithi164.stdout: x86_64 10:1.5.3-173.el7_8.3 @rhel-7-server-rpms 755 k 2020-07-15T08:57:05.697 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:32.254 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:python-rgw-14.2.10-197.g6feeb409f65.el7.x86_64 2/12 , 2020-07-15T07:02:32.523 INFO:teuthology.orchestra.run.smithi164.stdout: Erasing : 2:python-rados-14.2.10-197.g6feeb409f65.el7.x86_64 8/12 , 2020-07-15T07:02:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: Erasing : 2:python-rbd-14.2.10-197.g6feeb409f65.el7.x86_64 3/12 2020-07-15T08:57:05.698 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:38.094 INFO:teuthology.orchestra.run.smithi190.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- , 2020-07-15T07:02:38.095 INFO:teuthology.orchestra.run.smithi190.stdout: : disabled-repos, subscription-manager , 2020-07-15T07:02:38.386 INFO:teuthology.orchestra.run.smithi164.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2020-07-15T08:57:05.699 WARNING:scrape:Ignoring malparsed backtrace: 2020-07-15T07:02:42.013 INFO:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2020-07-15T07:02:42.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2020-07-15T07:02:42.102 INFO:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2020-07-15T08:57:05.721 INFO:scrape:Found 2 distinct failure reasons 2020-07-15T08:57:05.723 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 4000 --objects 300 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2020-07-15T08:57:05.723 INFO:scrape:1 jobs: ['5228306'] 2020-07-15T08:57:05.724 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log.yaml', '2-recovery-overrides/{default.yaml}', 'backoff/peering_and_degraded.yaml', 'ceph.yaml', 'clusters/{fixed-2.yaml', 'd-balancer/crush-compat.yaml', 'msgr-failures/few.yaml', 'msgr/simple.yaml', 'objectstore/bluestore-bitmap.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml', 'supported-random-distro$/{rhel_latest.yaml}', 'thrashers/pggrow.yaml', 'thrashosds-health.yaml', 'workloads/set-chunks-read.yaml}'] 2020-07-15T08:57:05.724 INFO:scrape: 2020-07-15T08:57:05.724 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2020-07-15T08:57:05.725 INFO:scrape:1 jobs: ['5228120'] 2020-07-15T08:57:05.725 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log.yaml', '2-recovery-overrides/{more-async-partial-recovery.yaml}', 'backoff/peering_and_degraded.yaml', 'ceph.yaml', 'clusters/{fixed-2.yaml', 'd-balancer/crush-compat.yaml', 'msgr-failures/few.yaml', 'msgr/random.yaml', 'objectstore/bluestore-comp-zlib.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml', 'supported-random-distro$/{ubuntu_latest.yaml}', 'thrashers/default.yaml', 'thrashosds-health.yaml', 'workloads/rados_api_tests.yaml}'] 2020-07-15T08:57:05.726 INFO:scrape: