2023-05-10T02:01:41.278 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-10T02:01:41.279 INFO:teuthology.results:subset: '344/1024' 2023-05-10T02:01:41.280 INFO:teuthology.results:seed: '1357' 2023-05-10T10:33:13.442 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-10T10:33:13.500 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-10T22:33:18.004 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-05-10T22:33:35.771 INFO:scrape:Found 200 jobs 2023-05-10T22:33:36.158 WARNING:scrape:Misunderstood line: 2023-05-10T22:33:39.079 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-3879-ge28f4bf0 (e28f4bf0df5d53a504fe40e32a78dd63506e2421) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fccef9616e1] , 2: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7fccef961895] 2023-05-10T22:33:39.080 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:51.314+0000 7f2418a12640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24141455a0 con 0x7f24140b1c30 , 2023-05-10T10:17:51.314+0000 7f24137fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140b0fb0 0x7f241406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:17:51.314+0000 7f24137fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140b0fb0 0x7f241406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36526/0 (socket says 172.21.15.92:36526) 2023-05-10T22:33:39.080 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 -- 172.21.15.92:0/4047004657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7aa01dc6f0 msgr2=0x7f7aa01dcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 --2- 172.21.15.92:0/4047004657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7aa01dc6f0 0x7f7aa01dcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 -- 172.21.15.92:0/4047004657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa00b7dc0 con 0x7f7a8c004d60 2023-05-10T22:33:39.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c08146010 0x7f2c08141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c0813f490 0x7f2c08140fb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c080b3700 0x7f2c08143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.082 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 -- start start , 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cd80b1300 0x7f9cd81468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f9cd8146e30 0x7f9cd814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.082 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 -- start start , 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f88cc069af0 0x7f88cc1398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88cc0b8450 0x7f88cc1ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.083 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.129 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster": { , 2023-05-10T10:17:53.130 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:53.130 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", 2023-05-10T22:33:39.083 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.149 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.150 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.150 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "0", , 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "6", , 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.085 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "how frequently to wake up and attempt optimization", , 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.085 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.086 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.248 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.249 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.249 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Prometheus container image", 2023-05-10T22:33:39.086 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: "daemon_cache_timeout": { , 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "daemon_cache_timeout", 2023-05-10T22:33:39.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "*", , 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.088 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.088 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "customized SSH config file to connect to managed hosts", , 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.089 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.387 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.388 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.089 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.407 INFO:teuthology.orchestra.run.smithi092.stdout: "ALERTMANAGER_API_HOST": { , 2023-05-10T10:17:53.408 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "ALERTMANAGER_API_HOST", , 2023-05-10T10:17:53.408 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", 2023-05-10T22:33:39.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.465 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:53.465 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.466 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-05-10T22:33:39.092 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.505 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_ADMIN_RESOURCE", , 2023-05-10T10:17:53.505 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.506 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.092 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.525 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.525 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:53.526 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.092 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: ], , 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", 2023-05-10T22:33:39.093 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.093 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.583 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.583 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.584 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.094 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.094 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.623 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: "interval": { 2023-05-10T22:33:39.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.661 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "port", , 2023-05-10T10:17:53.662 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2023-05-10T10:17:53.662 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: ], , 2023-05-10T10:17:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", 2023-05-10T22:33:39.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", 2023-05-10T22:33:39.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.779 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:53.780 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:53.780 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.099 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.818 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:53.819 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:53.819 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.099 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "3.0", 2023-05-10T22:33:39.100 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { , 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", 2023-05-10T22:33:39.100 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.877 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.877 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.878 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: "server_port": { 2023-05-10T22:33:39.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster_level": { 2023-05-10T22:33:39.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.955 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.955 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:53.956 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "roption2": { , 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "roption2", , 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.103 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.103 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.104 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.104 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.105 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.105 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.106 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:54.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.106 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-05-10T22:33:39.107 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [ 2023-05-10T22:33:39.107 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.171 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2023-05-10T10:17:54.171 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.172 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.107 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.108 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster": { , 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", 2023-05-10T22:33:39.108 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.229 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Password to authenticate with", , 2023-05-10T10:17:54.230 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.230 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "0", , 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "0", , 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "6", 2023-05-10T22:33:39.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.110 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-05-10T22:33:39.110 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", 2023-05-10T22:33:39.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Jaeger query container image", , 2023-05-10T10:17:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: "daemon_cache_timeout": { 2023-05-10T22:33:39.112 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.112 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.406 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.406 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "*", , 2023-05-10T10:17:54.407 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.113 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.113 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-05-10T22:33:39.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.465 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.465 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.466 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: "ALERTMANAGER_API_HOST": { , 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "ALERTMANAGER_API_HOST", 2023-05-10T22:33:39.115 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.504 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.504 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.505 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.115 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.117 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "RGW_API_ADMIN_RESOURCE": { , 2023-05-10T10:17:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_ADMIN_RESOURCE", , 2023-05-10T10:17:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.117 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.117 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.118 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.118 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.119 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.119 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.120 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.120 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.741 INFO:teuthology.orchestra.run.smithi092.stdout: "port": { , 2023-05-10T10:17:54.742 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "port", , 2023-05-10T10:17:54.742 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", 2023-05-10T22:33:39.120 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", 2023-05-10T22:33:39.121 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.780 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.780 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.781 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.121 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.122 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.122 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster_level", , 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.123 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.858 INFO:teuthology.orchestra.run.smithi092.stdout: "debug", , 2023-05-10T10:17:54.859 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:54.859 INFO:teuthology.orchestra.run.smithi092.stdout: "info", 2023-05-10T22:33:39.123 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.123 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.897 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2023-05-10T10:17:54.897 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:54.898 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", 2023-05-10T22:33:39.124 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "float", , 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.124 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { 2023-05-10T22:33:39.125 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.125 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.126 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.126 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "key_file": { , 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.127 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.127 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:55.056 INFO:teuthology.orchestra.run.smithi092.stdout: "roption2": { , 2023-05-10T10:17:55.057 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "roption2", 2023-05-10T22:33:39.127 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.076 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2023-05-10T10:17:55.076 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.077 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.128 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.096 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:55.096 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:55.097 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.128 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster_level", , 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.129 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.129 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { , 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", 2023-05-10T22:33:39.130 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.130 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", 2023-05-10T22:33:39.131 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.131 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.131 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "discovery_interval", , 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.132 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.132 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.291 INFO:teuthology.orchestra.run.smithi092.stdout: "status", , 2023-05-10T10:17:55.291 INFO:teuthology.orchestra.run.smithi092.stdout: "telemetry", , 2023-05-10T10:17:55.292 INFO:teuthology.orchestra.run.smithi092.stdout: "volumes" 2023-05-10T22:33:39.133 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.191631036, , 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000074474 , 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.133 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, , 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, , 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2023-05-10T22:33:39.134 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.502 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { , 2023-05-10T10:17:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, , 2023-05-10T10:17:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-05-10T22:33:39.134 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, , 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, , 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-05-10T22:33:39.135 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.542 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 478365, , 2023-05-10T10:17:55.543 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, , 2023-05-10T10:17:55.543 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-05-10T22:33:39.135 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 259024, , 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 51569210, , 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-05-10T22:33:39.135 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000000000, , 2023-05-10T10:17:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { , 2023-05-10T10:17:55.731 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2023-05-10T22:33:39.136 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, , 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, , 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 148823, 2023-05-10T22:33:39.138 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.910+0000 7feeec3a6640 1 -- 172.21.15.92:0/2862986098 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee40b1700 con 0x7feee4146a20 , 2023-05-10T10:17:55.910+0000 7feedaffd640 1 -- 172.21.15.92:0/2862986098 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7feedc00abd0 con 0x7feee4146a20 , 2023-05-10T10:17:55.910+0000 7feedaffd640 1 --2- 172.21.15.92:0/2862986098 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7feec8066280 0x7feec8068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.140 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:56.475 INFO:tasks.ceph.ceph_manager.ceph:clean! , 2023-05-10T10:17:56.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-05-10T10:17:56.642+0000 7f4d17412640 1 Processor -- start 2023-05-10T22:33:39.141 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:57.022+0000 7f8644952640 1 --2- 172.21.15.92:0/1910604831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86400bc060 0x7f864013f360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86300097c0 tx=0x7f863001c3c0 comp rx=0 tx=0).stop , 2023-05-10T10:17:57.022+0000 7f8644952640 1 -- 172.21.15.92:0/1910604831 shutdown_connections , 2023-05-10T10:17:57.022+0000 7f8644952640 1 --2- 172.21.15.92:0/1910604831 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f86400b3310 0x7f86400b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.141 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fcbe8004d60 0x7fcc000adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc001de330 0x7fcc00142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc00142b70 0x7fcc000acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.142 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:58.846+0000 7f7e83fff640 1 -- 172.21.15.92:0/4079243528 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e900b67a0 con 0x7f7e9c0892e0 , 2023-05-10T10:17:58.846+0000 7f7e83fff640 1 -- 172.21.15.92:0/4079243528 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900b6b50 con 0x7f7e9c0892e0 , 2023-05-10T10:17:58.846+0000 7f7ea1ed1640 1 -- 172.21.15.92:0/4079243528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e900084a0 con 0x7f7e9c0892e0 2023-05-10T22:33:39.142 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.194+0000 7f0674339640 1 --2- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6816/2447806576,v1:172.21.15.92:6817/2447806576] conn(0x7f065006ec00 0x7f0650071050 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:59.194+0000 7f0674339640 1 -- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f06500662b0 msgr2=0x7f0650068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:17:59.194+0000 7f0674339640 1 --2- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f06500662b0 0x7f0650068760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f066c1308b0 tx=0x7f0668008040 comp rx=0 tx=0).stop 2023-05-10T22:33:39.143 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.510+0000 7fab6df60640 1 -- 172.21.15.92:0/779489884 shutdown_connections , 2023-05-10T10:17:59.510+0000 7fab6df60640 1 -- 172.21.15.92:0/779489884 wait complete. , 2023-05-10T10:17:59.533 INFO:tasks.ceph:Scrubbing osd.2 2023-05-10T22:33:39.143 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740a58c0 con 0x7fd7840bbc60 , 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7740a5a20 con 0x7fd7840bbc60 , 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740a5db0 con 0x7fd7840bbc60 2023-05-10T22:33:39.144 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 -- 172.21.15.92:0/640647466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48180ba9b0 msgr2=0x7f48180b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 --2- 172.21.15.92:0/640647466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48180ba9b0 0x7f48180b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 -- 172.21.15.92:0/640647466 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810009a70 con 0x7f48180ba5e0 2023-05-10T22:33:39.144 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.382+0000 7fc607fff640 1 -- 172.21.15.92:0/3699793562 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7fc6100eb160 con 0x7fc6200ac150 , 2023-05-10T10:18:00.530+0000 7fc62621f640 1 -- 172.21.15.92:0/3699793562 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc62004dd50 con 0x7fc60c066280 , 2023-05-10T10:18:00.530+0000 7fc607fff640 1 -- 172.21.15.92:0/3699793562 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc62004dd50 con 0x7fc60c066280 2023-05-10T22:33:39.145 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.878+0000 7f87a8699640 1 Processor -- start , 2023-05-10T10:18:00.878+0000 7f87a8699640 1 -- start start , 2023-05-10T10:18:00.878+0000 7f87a8699640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f87a00b1ec0 0x7f87a013bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.146 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8904142aa0 0x7f8904145520 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f890413c2f0 0x7f8904142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> v2:172.21.15.194:3300/0 conn(0x7f89040b1ec0 0x7f890413bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.146 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.402+0000 7fae42d18640 1 --2- 172.21.15.92:0/190621659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae3c146e30 0x7fae3c1e0220 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c410 tx=0x7fae3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:18:01.406+0000 7fae32ffd640 1 -- 172.21.15.92:0/190621659 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae340074e0 con 0x7fae3c146e30 , 2023-05-10T10:18:01.406+0000 7fae32ffd640 1 -- 172.21.15.92:0/190621659 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34007640 con 0x7fae3c146e30 2023-05-10T22:33:39.147 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 -- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1b90066280 msgr2=0x7f1b90068730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 --2- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1b90066280 0x7f1b90068730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005680 tx=0x7f1ba0003dd0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 -- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb0145100 msgr2=0x7f1bb00b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.147 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a5950 con 0x7f3bec142960 , 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be40a5ab0 con 0x7f3bec142960 , 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a5e40 con 0x7f3bec142960 2023-05-10T22:33:39.148 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.406+0000 7fbafce28640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbaf80b1ec0 0x7fbaf80b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:44774/0 (socket says 172.21.15.92:44774) , 2023-05-10T10:18:02.406+0000 7fbafce28640 1 -- 172.21.15.92:0/4219600763 learned_addr learned my addr 172.21.15.92:0/4219600763 (peer_addr_for_me v2:172.21.15.92:0/0) , 2023-05-10T10:18:02.406+0000 7fbaeffff640 1 --2- 172.21.15.92:0/4219600763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbaf8142560 0x7fbaf81e0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.149 WARNING:scrape:Ignoring malparsed backtrace: dumped all , 2023-05-10T10:18:02.894+0000 7f1236f98640 1 -- 172.21.15.92:0/4289600361 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f12140662b0 msgr2=0x7f1214068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:02.894+0000 7f1236f98640 1 --2- 172.21.15.92:0/4289600361 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f12140662b0 0x7f1214068760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1224007fd0 tx=0x7f1224009040 comp rx=0 tx=0).stop 2023-05-10T22:33:39.151 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.151 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:23.299 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:23.299 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:23.300 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.153 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:38.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-10T10:18:43.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-05-10T10:18:43.469+0000 7f59ae939640 1 Processor -- start 2023-05-10T22:33:39.153 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:43.679 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:43.679 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:43.680 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.154 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 --2- 172.21.15.92:0/1761086283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7f8004d10 0x7fb7f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 -- 172.21.15.92:0/1761086283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bb20 con 0x7fb7f80040a0 , 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 -- 172.21.15.92:0/1761086283 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bc60 con 0x7fb7f8004d10 2023-05-10T22:33:39.155 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:04.049 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:04.050 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:04.050 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.156 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:24.213+0000 7f01d088c640 1 -- start start , 2023-05-10T10:19:24.213+0000 7f01d088c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01cc0b40c0 0x7f01cc13cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:19:24.213+0000 7f01d088c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01cc0b5610 0x7f01cc13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.157 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:24.424 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:24.425 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:24.425 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.158 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:44.593+0000 7f3accff9640 1 -- 172.21.15.92:0/861657520 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7f3ab80b36f0 con 0x7f3ad00bc060 , 2023-05-10T10:19:44.737+0000 7f3ad58d1640 1 -- 172.21.15.92:0/861657520 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ad004dd50 con 0x7f3ab0066280 , 2023-05-10T10:19:44.745+0000 7f3accff9640 1 -- 172.21.15.92:0/861657520 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312088 (secure 0 0 0) 0x7f3ad004dd50 con 0x7f3ab0066280 2023-05-10T22:33:39.160 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:44.860 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:44.861 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:44.861 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.160 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:05.210 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:05.210 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:05.211 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.162 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:25.376+0000 7f3990a11640 1 --2- 172.21.15.92:0/3688155896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f398c13c5a0 0x7f398c14ca50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f398c0b19c0 tx=0x7f398401eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291d182945c6a452 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:20:25.376+0000 7f398affd640 1 -- 172.21.15.92:0/3688155896 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a7830 con 0x7f398c13c5a0 , 2023-05-10T10:20:25.376+0000 7f398affd640 1 -- 172.21.15.92:0/3688155896 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840a7990 con 0x7f398c13c5a0 2023-05-10T22:33:39.162 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.164 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:45.760+0000 7fbec659c640 1 --2- 172.21.15.92:0/1247818276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbec80b0be0 0x7fbec8144310 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800e870 tx=0x7fbeb800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:20:45.760+0000 7fbebf7fe640 1 -- 172.21.15.92:0/1247818276 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8009510 con 0x7fbec80b0be0 , 2023-05-10T10:20:45.760+0000 7fbebf7fe640 1 -- 172.21.15.92:0/1247818276 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb800eec0 con 0x7fbec80b0be0 2023-05-10T22:33:39.165 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:45.981 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:45.981 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:45.982 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.166 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:06.334 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:06.335 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:06.335 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.167 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:26.504+0000 7fde10f29640 1 -- 172.21.15.92:0/3502791245 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc0a7050 con 0x7fde0c0af9f0 , 2023-05-10T10:21:26.504+0000 7fde10f29640 1 -- 172.21.15.92:0/3502791245 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0ab980 con 0x7fde0c0af9f0 , 2023-05-10T10:21:26.504+0000 7fde1418f640 1 -- 172.21.15.92:0/3502791245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde0c0af9f0 msgr2=0x7fde0c149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.167 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.169 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:46.876+0000 7ff8ad7fa640 1 -- 172.21.15.92:0/1067990458 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800f040 con 0x7ff8b0146a20 , 2023-05-10T10:21:46.876+0000 7ff8b6d25640 1 -- 172.21.15.92:0/1067990458 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b006f2a0 con 0x7ff8b0146a20 , 2023-05-10T10:21:46.876+0000 7ff8ad7fa640 1 -- 172.21.15.92:0/1067990458 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800de80 con 0x7ff8b0146a20 2023-05-10T22:33:39.169 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:47.089 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:47.089 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:47.090 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.171 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:07.407+0000 7fa723b25640 1 --2- 172.21.15.92:0/1033317209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa71c0b29e0 0x7fa71c1d6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:22:07.407+0000 7fa723b25640 1 --2- 172.21.15.92:0/1033317209 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa708006f20 0x7fa71c1450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:22:07.407+0000 7fa723b25640 1 -- 172.21.15.92:0/1033317209 >> 172.21.15.92:0/1033317209 conn(0x7fa71c0b73c0 msgr2=0x7fa71c0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-10T22:33:39.172 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4003530 con 0x7fa6cc13c1d0 , 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986438495 0 0) 0x7fa6b4003530 con 0x7fa6cc13c1d0 , 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 >> v2:172.21.15.194:3300/0 conn(0x7fa6cc0afdf0 msgr2=0x7fa6cc1456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.173 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:27.815 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:27.815 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:27.816 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.174 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:47.999+0000 7f9e0b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c13b710 0x7f9e0c1e0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48960/0 (socket says 172.21.15.92:48960) , 2023-05-10T10:22:47.999+0000 7f9e0b7fe640 1 -- 172.21.15.92:0/4248290134 learned_addr learned my addr 172.21.15.92:0/4248290134 (peer_addr_for_me v2:172.21.15.92:0/0) , 2023-05-10T10:22:47.999+0000 7f9e0affd640 1 --2- 172.21.15.92:0/4248290134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e0c0b0250 0x7f9e0c12e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.175 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.176 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 --2- 172.21.15.92:0/2013621284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f115413d6d0 0x7f11541df230 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f114800f5a0 tx=0x7f114800f690 comp rx=0 tx=0).stop , 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 -- 172.21.15.92:0/2013621284 shutdown_connections , 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 --2- 172.21.15.92:0/2013621284 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f11541df770 0x7f11541dfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.177 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:28.734+0000 7f7fe5ffb640 1 -- 172.21.15.92:0/3273409605 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706832390 0 0) 0x7f7fe80b8d40 con 0x7f7fe8142860 , 2023-05-10T10:23:28.734+0000 7f7fe5ffb640 1 -- 172.21.15.92:0/3273409605 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe80b8ad0 con 0x7f7fe8142860 , 2023-05-10T10:23:28.734+0000 7f7fe67fc640 1 -- 172.21.15.92:0/3273409605 >> v1:172.21.15.194:6789/0 conn(0x7f7fe80b83a0 legacy=0x7f7fe806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.178 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:28.941 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:23:28.942 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:23:28.942 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.179 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:49.269+0000 7f3e09ffb640 1 -- 172.21.15.92:0/2342637346 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+313789 (secure 0 0 0) 0x7f3e1404dd50 con 0x7f3df80662b0 , 2023-05-10T10:23:49.269 INFO:teuthology.orchestra.run.smithi092.stdout: , dumped all 2023-05-10T22:33:39.181 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:09.465+0000 7fa5360ef640 1 --2- 172.21.15.92:0/1384444669 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa53013c5a0 0x7fa53014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:24:09.465+0000 7fa5348ec640 1 -- 172.21.15.92:0/1384444669 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514500000 0 0) 0x7fa530071ac0 con 0x7fa53013c1d0 , 2023-05-10T10:24:09.465+0000 7fa5348ec640 1 -- 172.21.15.92:0/1384444669 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c003530 con 0x7fa53013c1d0 2023-05-10T22:33:39.181 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.183 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 -- 172.21.15.92:0/3004979523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c00146a20 msgr2=0x7f6c001e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 --2- 172.21.15.92:0/3004979523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c00146a20 0x7f6c001e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 -- 172.21.15.92:0/3004979523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0009420 con 0x7f6c0014bac0 2023-05-10T22:33:39.184 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:30.045 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:30.045 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:30.046 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.185 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.186 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af006bed0 con 0x7f5af00b40c0 , 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af006bd40 con 0x7f5af00b40c0 , 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af00b5c80 con 0x7f5af00b40c0 2023-05-10T22:33:39.187 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:30.926+0000 7fed25012640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fed200b1c30 0x7fed200b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:25:30.926+0000 7fed25012640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed2013f400 0x7fed2013f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:25:30.926+0000 7fed25012640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed200b38c0 con 0x7fed200b0fb0 2023-05-10T22:33:39.188 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.189 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:51.297+0000 7f44f37fe640 1 -- 172.21.15.92:0/1921343202 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f44ec083d80 con 0x7f44fc0b0be0 , 2023-05-10T10:25:51.297+0000 7f44f1d9b640 1 --2- 172.21.15.92:0/1921343202 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f44dc0662b0 0x7f44dc068760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:25:51.297+0000 7f44f1d9b640 1 --2- 172.21.15.92:0/1921343202 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f44dc0662b0 0x7f44dc068760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44e40af000 tx=0x7f44e40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:11.652+0000 7f38bffff640 1 --2- 172.21.15.92:0/2226374949 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f38c00bc060 0x7f38c013f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:26:11.652+0000 7f38beffd640 1 -- 172.21.15.92:0/2226374949 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430380296 0 0) 0x7f38c008a060 con 0x7f38c013f8a0 , 2023-05-10T10:26:11.652+0000 7f38beffd640 1 -- 172.21.15.92:0/2226374949 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a8003530 con 0x7f38c013f8a0 2023-05-10T22:33:39.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.193 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:32.020+0000 7f2826ffd640 1 -- 172.21.15.92:0/3939449709 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820004020 con 0x7f283006ed00 , 2023-05-10T10:26:32.020+0000 7f2838360640 1 -- 172.21.15.92:0/3939449709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28300b2150 con 0x7f283006ed00 , 2023-05-10T10:26:32.020+0000 7f2826ffd640 1 -- 172.21.15.92:0/3939449709 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28200043b0 con 0x7f283006ed00 2023-05-10T22:33:39.194 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:52.379+0000 7f19997f0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1994145dd0 con 0x7f199413c5a0 , 2023-05-10T10:26:52.379+0000 7f19997f0640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19940b3c10 con 0x7f199413c1d0 , 2023-05-10T10:26:52.379+0000 7f19937fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f199413c5a0 0x7f199414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.195 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:52.580 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:52.580 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:52.581 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.196 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:12.742+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a480b2150 con 0x7f3a4813cba0 , 2023-05-10T10:27:12.742+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a480ae990 con 0x7f3a4813cba0 , 2023-05-10T10:27:12.746+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4804dd50 con 0x7f3a4813cba0 2023-05-10T22:33:39.197 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:33.094+0000 7f390659c640 1 --2- 172.21.15.92:0/2968357658 >> v2:172.21.15.194:3300/0 conn(0x7f39080afdf0 0x7f39081456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:27:33.094+0000 7f390759e640 1 --2- 172.21.15.92:0/2968357658 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f390813c5a0 0x7f39080ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:27:33.094+0000 7f3906d9d640 1 -- 172.21.15.92:0/2968357658 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f390813c5a0 msgr2=0x7f39080ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.198 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:33.298 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:27:33.299 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:27:33.299 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.199 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 -- 172.21.15.92:0/3384285672 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7f7de80067e0 con 0x7f7de00039b0 , 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 --2- 172.21.15.92:0/3384285672 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f7de4066280 0x7f7de4068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 -- 172.21.15.92:0/3384285672 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f7de811e330 con 0x7f7de00039b0 2023-05-10T22:33:39.201 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:13.825+0000 7fb39898c640 1 -- 172.21.15.92:0/4118702217 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39406e570 con 0x7fb39413c1d0 , 2023-05-10T10:28:13.829+0000 7fb39898c640 1 --2- 172.21.15.92:0/4118702217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb39413c1d0 0x7fb3940b1d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb3940b13c0 tx=0x7fb39001eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c990e17de74807ab server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:28:13.829+0000 7fb38b7fe640 1 -- 172.21.15.92:0/4118702217 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900a79d0 con 0x7fb39413c1d0 2023-05-10T22:33:39.202 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:14.032 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:14.032 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:14.033 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.203 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:34.381 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:34.381 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:34.382 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.204 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58940052d0 con 0x7f58a00b07d0 , 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894005430 con 0x7f58a00b07d0 , 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894012870 con 0x7f58a00b07d0 2023-05-10T22:33:39.205 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 -- 172.21.15.92:0/851029519 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187886880 0 0) 0x7f53b0003530 con 0x7f53c413c1d0 , 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 -- 172.21.15.92:0/851029519 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f53c413c5a0 msgr2=0x7f53c414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 --2- 172.21.15.92:0/851029519 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f53c413c5a0 0x7f53c414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.206 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:15.120 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:29:15.120 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:29:15.121 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.207 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 -- 172.21.15.92:0/3083124976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4dd00b5610 msgr2=0x7f4dd0139a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 --2- 172.21.15.92:0/3083124976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4dd00b5610 0x7f4dd0139a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0019150 tx=0x7f4dc001ad80 comp rx=0 tx=0).stop , 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 -- 172.21.15.92:0/3083124976 shutdown_connections 2023-05-10T22:33:39.209 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43c0afdf0 0x7fa43c142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43c0afdf0 0x7fa43c142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42862/0 (socket says 172.21.15.92:42862) , 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 -- 172.21.15.92:0/1418296414 learned_addr learned my addr 172.21.15.92:0/1418296414 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-05-10T22:33:39.210 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f3c0b40a0 0x7f9f3c147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f3c0b40a0 0x7f9f3c147240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47766/0 (socket says 172.21.15.92:47766) , 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 -- 172.21.15.92:0/1565947473 learned_addr learned my addr 172.21.15.92:0/1565947473 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-05-10T22:33:39.211 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.212 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 --2- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f02940662b0 0x7f0294068760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f02a80b2920 tx=0x7f029c008040 comp rx=0 tx=0).stop , 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 -- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0290004d10 msgr2=0x7f02a8143380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 --2- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0290004d10 0x7f02a8143380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f029800f810 tx=0x7f029800fcd0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.214 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:56.721+0000 7f0c20c0e640 1 -- 172.21.15.92:0/450291203 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c18005a60 con 0x7f0c08003b10 , 2023-05-10T10:30:56.721+0000 7f0c20c0e640 1 -- 172.21.15.92:0/450291203 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18005e50 con 0x7f0c08003b10 , 2023-05-10T10:30:56.725+0000 7f0c22c12640 1 -- 172.21.15.92:0/450291203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c08003b10 msgr2=0x7f0c08003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.215 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:56.938 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:56.939 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2023-05-10T10:31:12.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-10T22:33:39.217 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:17.288 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:17.288 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:17.289 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.217 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.219 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:57.811+0000 7f190d2d5640 1 --2- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18f4003b10 0x7f1908145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:31:57.811+0000 7f190dad6640 1 --2- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f19080b51a0 0x7f19080b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:31:57.811+0000 7f190cad4640 1 -- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f19080b51a0 msgr2=0x7f19080b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.220 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:18.167+0000 7f09f4855640 1 --2- 172.21.15.92:0/1532289199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b2ab0 0x7f09f01420d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09d800a960 tx=0x7f09d801d7c0 comp rx=0 tx=0).stop , 2023-05-10T10:32:18.167+0000 7f09f4855640 1 -- 172.21.15.92:0/1532289199 shutdown_connections , 2023-05-10T10:32:18.167+0000 7f09f4855640 1 --2- 172.21.15.92:0/1532289199 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f09f013b710 0x7f09f014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.222 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:38.515+0000 7fecf54f9640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006e7b0 con 0x7fecf00b0be0 , 2023-05-10T10:32:38.515+0000 7fecee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecf0146a20 0x7fecf0146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:32:38.515+0000 7fecee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecf0146a20 0x7fecf0146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53002/0 (socket says 172.21.15.92:53002) 2023-05-10T22:33:39.223 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:38.718 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:38.718 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:38.719 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.224 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:59.071 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:59.072 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:59.072 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.225 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:19.418 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:33:19.418 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:33:19.419 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.226 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 >> 172.21.15.92:0/1999965888 conn(0x7fe6600b73c0 msgr2=0x7fe6600b1c40 unknown :-1 s=STATE_NONE l=0).mark_down , 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 shutdown_connections , 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 wait complete. 2023-05-10T22:33:39.228 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:59.937+0000 7fee48ff9640 1 -- 172.21.15.92:0/1818378791 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44001210 con 0x7fee4c150b50 , 2023-05-10T10:33:59.937+0000 7fee50a12640 1 -- 172.21.15.92:0/1818378791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c151550 con 0x7fee4c150b50 , 2023-05-10T10:33:59.941+0000 7fee50a12640 1 -- 172.21.15.92:0/1818378791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c151550 con 0x7fee4c150b50 2023-05-10T22:33:39.229 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:34:20.288+0000 7f5664ff9640 1 -- 172.21.15.92:0/3694622742 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660007640 con 0x7f566814bac0 , 2023-05-10T10:34:20.288+0000 7f5664ff9640 1 -- 172.21.15.92:0/3694622742 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660004a30 con 0x7f566814bac0 , 2023-05-10T10:34:20.288+0000 7f566db6d640 1 -- 172.21.15.92:0/3694622742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56680b1700 con 0x7f566814bac0 2023-05-10T22:33:39.231 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:34:40.647+0000 7f8468ff9640 1 -- 172.21.15.92:0/2348478847 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8460005430 con 0x7f846c0af620 , 2023-05-10T10:34:40.647+0000 7f8468ff9640 1 -- 172.21.15.92:0/2348478847 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460003820 con 0x7f846c0af620 , 2023-05-10T10:34:40.647+0000 7f8472841640 1 -- 172.21.15.92:0/2348478847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c149fe0 con 0x7f846c0af620 2023-05-10T22:33:39.232 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:00.998+0000 7fad51708640 1 -- 172.21.15.92:0/1839056521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48009a70 con 0x7fad4c13c1d0 , 2023-05-10T10:35:00.998+0000 7fad5270a640 1 --2- 172.21.15.92:0/1839056521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad4c13c5a0 0x7fad4c1e58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! , 2023-05-10T10:35:00.998+0000 7fad51708640 1 --2- 172.21.15.92:0/1839056521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad4c13c1d0 0x7fad4c1e53a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e840 tx=0x7fad3c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.234 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:21.354+0000 7f39b5117640 1 --2- 172.21.15.92:0/3488893550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b013c5a0 0x7f39b0134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:35:21.354+0000 7f39b5117640 1 -- 172.21.15.92:0/3488893550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac009a70 con 0x7f39b00afdf0 , 2023-05-10T10:35:21.354+0000 7f39b4916640 1 --2- 172.21.15.92:0/3488893550 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f39b013c1d0 0x7f39b014c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-10T22:33:39.235 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:41.705+0000 7ff56146d640 1 --2- 172.21.15.92:0/3276553125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff55c1494a0 0x7ff55c1ddac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:35:41.705+0000 7ff56146d640 1 -- 172.21.15.92:0/3276553125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c009420 con 0x7ff55c0abda0 , 2023-05-10T10:35:41.705+0000 7ff56146d640 1 --2- 172.21.15.92:0/3276553125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c0abda0 0x7ff55c144ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff55800e930 tx=0x7ff55800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.236 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:02.052+0000 7fca8b798640 1 -- 172.21.15.92:0/1107612113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8406f2a0 con 0x7fca840b0be0 , 2023-05-10T10:36:02.052+0000 7fca8b798640 1 -- 172.21.15.92:0/1107612113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca840b1700 con 0x7fca840b0be0 , 2023-05-10T10:36:02.056+0000 7fca79ffb640 1 -- 172.21.15.92:0/1107612113 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7fca800073d0 con 0x7fca840b0be0 2023-05-10T22:33:39.238 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:22.412+0000 7fd9a6ffd640 1 -- 172.21.15.92:0/1005505257 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7fd9a00eb540 con 0x7fd9b013c1d0 , 2023-05-10T10:36:22.560+0000 7fd9b8325640 1 -- 172.21.15.92:0/1005505257 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9b013f380 con 0x7fd994066280 , 2023-05-10T10:36:22.568+0000 7fd9a6ffd640 1 -- 172.21.15.92:0/1005505257 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+317459 (secure 0 0 0) 0x7fd9ac005080 con 0x7fd994066280 2023-05-10T22:33:39.239 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 --2- 172.21.15.92:0/1529654815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f9c071c40 0x7f9f9c06f5b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800c490 tx=0x7f9f9800c950 comp rx=0 tx=0).stop , 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 -- 172.21.15.92:0/1529654815 shutdown_connections , 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 --2- 172.21.15.92:0/1529654815 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f9f800662b0 0x7f9f80068760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.242 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 >> 172.21.15.92:0/2541361664 conn(0x7f170406ba10 msgr2=0x7f17040b6a30 unknown :-1 s=STATE_NONE l=0).mark_down , 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 shutdown_connections , 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 wait complete. 2023-05-10T22:33:39.243 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:37:23.647 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:37:23.648 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:37:23.648 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.245 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:04.145+0000 7f3127462640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f31200b0fb0 0x7f31200b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:38:04.145+0000 7f3127462640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31200b38c0 con 0x7f31200b1c30 , 2023-05-10T10:38:04.145+0000 7f3127462640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312006df80 con 0x7f312013f400 2023-05-10T22:33:39.246 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:24.492+0000 7f5ba4b69640 1 --2- 172.21.15.92:0/3454878715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba00afdf0 0x7f5ba01456c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5b900097c0 tx=0x7f5b9001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19728d4d796bdca8 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:38:24.492+0000 7f5b97fff640 1 -- 172.21.15.92:0/3454878715 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b900ac640 con 0x7f5ba00afdf0 , 2023-05-10T10:38:24.492+0000 7f5b97fff640 1 -- 172.21.15.92:0/3454878715 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b900ac7a0 con 0x7f5ba00afdf0 2023-05-10T22:33:39.247 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:44.844+0000 7f3d49812640 1 -- 172.21.15.92:0/2590519533 wait complete. , 2023-05-10T10:38:44.844+0000 7f3d49812640 1 Processor -- start , 2023-05-10T10:38:44.844+0000 7f3d49812640 1 -- start start 2023-05-10T22:33:39.249 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:05.195+0000 7f8b8e59c640 1 --2- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b9813c1d0 0x7f8b98143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:39:05.195+0000 7f8b9759e640 1 --2- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8b9813c5a0 0x7f8b98143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:39:05.195+0000 7f8b96d9d640 1 -- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8b9813c5a0 msgr2=0x7f8b98143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.250 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:25.554+0000 7f109ffff640 1 -- 172.21.15.92:0/1930888513 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7f10900baba0 con 0x7f10a813c1d0 , 2023-05-10T10:39:25.554+0000 7f109ffff640 1 --2- 172.21.15.92:0/1930888513 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1088066280 0x7f1088068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:39:25.554+0000 7f109ffff640 1 -- 172.21.15.92:0/1930888513 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f109011f130 con 0x7f10a813c1d0 2023-05-10T22:33:39.252 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7faafc066280 0x7faafc068730 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab18146a20 0x7fab18070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fab180b0be0 0x7fab180715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.254 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:40:26.572+0000 7f84cc2fc640 1 -- start start , 2023-05-10T10:40:26.576+0000 7f84cc2fc640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f84c413c1d0 0x7f84c40b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:40:26.576+0000 7f84cc2fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f84c40afdf0 0x7f84c41456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.255 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:40:46.923+0000 7ff0b1336640 1 -- 172.21.15.92:0/998423516 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267002128 0 0) 0x7ff0ac0b17f0 con 0x7ff0ac0af620 , 2023-05-10T10:40:46.923+0000 7ff0b2b39640 1 --2- 172.21.15.92:0/998423516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ac071c40 0x7ff0ac149c80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac073770 tx=0x7ff0a401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d3eb06948a13b9 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:40:46.923+0000 7ff0b1336640 1 -- 172.21.15.92:0/998423516 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40ab600 con 0x7ff0ac071c40 2023-05-10T22:33:39.257 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:07.270+0000 7f5622525640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b16f0 con 0x7f561c0b3310 , 2023-05-10T10:41:07.270+0000 7f561bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0aee10 0x7f561c13dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:41:07.270+0000 7f561bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0aee10 0x7f561c13dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33418/0 (socket says 172.21.15.92:33418) 2023-05-10T22:33:39.258 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:27.613+0000 7f822b539640 1 -- 172.21.15.92:0/2379905546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822414d6d0 con 0x7f8224071c40 , 2023-05-10T10:41:27.613+0000 7f8219ffb640 1 -- 172.21.15.92:0/2379905546 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000ce30 con 0x7f8224071c40 , 2023-05-10T10:41:27.613+0000 7f822b539640 1 -- 172.21.15.92:0/2379905546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822414d540 con 0x7f8224071c40 2023-05-10T22:33:39.260 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:48.116+0000 7fa70de41640 1 -- 172.21.15.92:0/246480377 shutdown_connections , 2023-05-10T10:41:48.116+0000 7fa70de41640 1 --2- 172.21.15.92:0/246480377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa70813c5a0 0x7fa70813b090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:41:48.116+0000 7fa70de41640 1 --2- 172.21.15.92:0/246480377 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa70813c1d0 0x7fa708138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.262 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:42:28.678+0000 7f3ba71a2640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba006df80 con 0x7f3ba00b0fb0 , 2023-05-10T10:42:28.678+0000 7f3ba71a2640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba006e260 con 0x7f3ba00b1c30 , 2023-05-10T10:42:28.678+0000 7f3ba69a1640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3ba013f400 0x7f3ba014c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57762/0 (socket says 172.21.15.92:57762) 2023-05-10T22:33:39.263 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:42:49.017+0000 7f080a59c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f080c0b0be0 0x7f080c13d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49914/0 (socket says 172.21.15.92:49914) , 2023-05-10T10:42:49.017+0000 7f080a59c640 1 -- 172.21.15.92:0/2078512857 learned_addr learned my addr 172.21.15.92:0/2078512857 (peer_addr_for_me v1:172.21.15.92:0/0) , 2023-05-10T10:42:49.017+0000 7f0809d9b640 1 -- 172.21.15.92:0/2078512857 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767686239 0 0) 0x7f080c13d8d0 con 0x7f080c0b0be0 2023-05-10T22:33:39.265 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:09.357+0000 7fc6f96d6640 1 --2- 172.21.15.92:0/4265691674 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc6f413c5a0 0x7fc6f414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:43:09.361+0000 7fc6f8ed5640 1 --2- 172.21.15.92:0/4265691674 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc6f413c5a0 0x7fc6f414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:09.361+0000 7fc6f8ed5640 1 -- 172.21.15.92:0/4265691674 >> v1:172.21.15.92:6789/0 conn(0x7fc6f40afdf0 legacy=0x7fc6f41456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.266 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:29.700+0000 7fe73d569640 1 --2- 172.21.15.92:0/1350475399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe724003980 0x7fe724003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:29.700+0000 7fe73d569640 1 -- 172.21.15.92:0/1350475399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724003530 con 0x7fe7240047d0 , 2023-05-10T10:43:29.700+0000 7fe73e56b640 1 --2- 172.21.15.92:0/1350475399 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe724005380 0x7fe72400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-10T22:33:39.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728142650 con 0x7f97280bc060 , 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728139c10 con 0x7f97280b3310 , 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972813b830 con 0x7f972813f8a0 2023-05-10T22:33:39.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:50.419+0000 7f3e2bfff640 1 -- 172.21.15.92:0/1064163243 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f3e24084240 con 0x7f3e340b1ec0 , 2023-05-10T10:43:50.419+0000 7f3e2a7fc640 1 --2- 172.21.15.92:0/1064163243 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f3e14066280 0x7f3e14068730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:43:50.419+0000 7f3e2a7fc640 1 --2- 172.21.15.92:0/1064163243 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f3e14066280 0x7f3e14068730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c006fd0 tx=0x7f3e1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.269 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.083+0000 7f47cd18f640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f47c813c1d0 0x7f47c80b1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:52234/0 (socket says 172.21.15.92:52234) , 2023-05-10T10:43:51.083+0000 7f47cd18f640 1 -- 172.21.15.92:0/3437986477 learned_addr learned my addr 172.21.15.92:0/3437986477 (peer_addr_for_me v1:172.21.15.92:0/0) , 2023-05-10T10:43:51.083+0000 7f47cc98e640 1 --2- 172.21.15.92:0/3437986477 >> v2:172.21.15.92:3300/0 conn(0x7f47c80afdf0 0x7f47c81456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.427+0000 7fe239a95640 1 -- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe2340b0250 msgr2=0x7fe2200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:43:51.427+0000 7fe239a95640 1 --2- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe2340b0250 0x7fe2200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:51.427+0000 7fe239a95640 1 -- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe220004d10 msgr2=0x7fe220005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 -- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff398146e30 msgr2=0x7ff3981e1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 --2- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff398146e30 0x7ff3981e1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 -- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3980b1300 msgr2=0x7ff3981e0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fad68145600 0x7fad681e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad680b7a20 0x7fad681e0a40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7fad54066280 0x7fad54068730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 -- 172.21.15.92:0/3107303740 >> v2:172.21.15.194:3300/0 conn(0x7f0e580b8140 msgr2=0x7f0e580b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 --2- 172.21.15.92:0/3107303740 >> v2:172.21.15.194:3300/0 conn(0x7f0e580b8140 0x7f0e580b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 -- 172.21.15.92:0/3107303740 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58148cd0 con 0x7f0e580ba5e0 2023-05-10T22:33:39.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:44:20.439 INFO:tasks.ceph.osd.6:Stopped , 2023-05-10T10:44:20.440 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2023-05-10T10:44:20.440 INFO:teuthology.orchestra.run:waiting for 300 2023-05-10T22:33:39.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== , 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:*hv01.front.sepi 44.4.53.2 2 u 13 64 377 0.066 -3.000 2.285 , 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 64 377 0.089 -0.413 3.395 2023-05-10T22:33:39.316 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-3879-ge28f4bf0 (e28f4bf0df5d53a504fe40e32a78dd63506e2421) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fccef9616e1] , 2: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7fccef961895] 2023-05-10T22:33:39.317 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:51.314+0000 7f2418a12640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24141455a0 con 0x7f24140b1c30 , 2023-05-10T10:17:51.314+0000 7f24137fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140b0fb0 0x7f241406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:17:51.314+0000 7f24137fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140b0fb0 0x7f241406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36526/0 (socket says 172.21.15.92:36526) 2023-05-10T22:33:39.317 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 -- 172.21.15.92:0/4047004657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7aa01dc6f0 msgr2=0x7f7aa01dcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 --2- 172.21.15.92:0/4047004657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7aa01dc6f0 0x7f7aa01dcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 -- 172.21.15.92:0/4047004657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa00b7dc0 con 0x7f7a8c004d60 2023-05-10T22:33:39.318 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c08146010 0x7f2c08141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c0813f490 0x7f2c08140fb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c080b3700 0x7f2c08143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.318 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 -- start start , 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cd80b1300 0x7f9cd81468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f9cd8146e30 0x7f9cd814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.319 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 -- start start , 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f88cc069af0 0x7f88cc1398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88cc0b8450 0x7f88cc1ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.319 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.129 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster": { , 2023-05-10T10:17:53.130 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:53.130 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", 2023-05-10T22:33:39.320 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.149 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.150 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.150 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.320 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "0", , 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "6", , 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.321 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.321 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "how frequently to wake up and attempt optimization", , 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.322 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.322 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.248 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.249 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.249 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Prometheus container image", 2023-05-10T22:33:39.323 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.323 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: "daemon_cache_timeout": { , 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "daemon_cache_timeout", 2023-05-10T22:33:39.323 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.324 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "*", , 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.324 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.325 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "customized SSH config file to connect to managed hosts", , 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.325 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.387 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.388 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.326 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.407 INFO:teuthology.orchestra.run.smithi092.stdout: "ALERTMANAGER_API_HOST": { , 2023-05-10T10:17:53.408 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "ALERTMANAGER_API_HOST", , 2023-05-10T10:17:53.408 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.326 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", 2023-05-10T22:33:39.326 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.327 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.465 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:53.465 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.466 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.327 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-05-10T22:33:39.328 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.505 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_ADMIN_RESOURCE", , 2023-05-10T10:17:53.505 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.506 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.328 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.525 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.525 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:53.526 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.329 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: ], , 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", 2023-05-10T22:33:39.329 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.330 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.583 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.583 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.584 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.330 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.330 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.623 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.331 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: "interval": { 2023-05-10T22:33:39.331 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.661 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "port", , 2023-05-10T10:17:53.662 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2023-05-10T10:17:53.662 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.332 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.332 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: ], , 2023-05-10T10:17:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", 2023-05-10T22:33:39.333 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", 2023-05-10T22:33:39.333 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.333 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.334 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.779 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:53.780 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:53.780 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.334 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.335 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.818 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:53.819 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:53.819 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.335 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "3.0", 2023-05-10T22:33:39.336 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { , 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", 2023-05-10T22:33:39.336 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.877 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.877 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.878 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.336 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: "server_port": { 2023-05-10T22:33:39.337 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster_level": { 2023-05-10T22:33:39.337 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.338 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.955 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.955 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:53.956 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.338 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "roption2": { , 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "roption2", , 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.339 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.339 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.340 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.340 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.340 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.341 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.341 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:54.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.342 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-05-10T22:33:39.342 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [ 2023-05-10T22:33:39.343 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.171 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2023-05-10T10:17:54.171 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.172 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.343 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.343 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster": { , 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", 2023-05-10T22:33:39.344 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.229 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Password to authenticate with", , 2023-05-10T10:17:54.230 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.230 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.344 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "0", , 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "0", , 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "6", 2023-05-10T22:33:39.345 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.345 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-05-10T22:33:39.346 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", 2023-05-10T22:33:39.346 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.347 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Jaeger query container image", , 2023-05-10T10:17:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.347 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: "daemon_cache_timeout": { 2023-05-10T22:33:39.347 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.348 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.406 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.406 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "*", , 2023-05-10T10:17:54.407 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.348 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.349 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-05-10T22:33:39.349 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.465 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.465 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.466 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.350 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: "ALERTMANAGER_API_HOST": { , 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "ALERTMANAGER_API_HOST", 2023-05-10T22:33:39.350 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.504 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.504 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.505 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.350 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.351 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.351 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.352 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "RGW_API_ADMIN_RESOURCE": { , 2023-05-10T10:17:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_ADMIN_RESOURCE", , 2023-05-10T10:17:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.352 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.353 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.353 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.353 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.354 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.354 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.355 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.355 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.741 INFO:teuthology.orchestra.run.smithi092.stdout: "port": { , 2023-05-10T10:17:54.742 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "port", , 2023-05-10T10:17:54.742 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", 2023-05-10T22:33:39.356 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", 2023-05-10T22:33:39.356 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.780 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.780 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.781 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.356 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.357 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.357 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster_level", , 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.358 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.858 INFO:teuthology.orchestra.run.smithi092.stdout: "debug", , 2023-05-10T10:17:54.859 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:54.859 INFO:teuthology.orchestra.run.smithi092.stdout: "info", 2023-05-10T22:33:39.358 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.359 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.897 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2023-05-10T10:17:54.897 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:54.898 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", 2023-05-10T22:33:39.359 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "float", , 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.359 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { 2023-05-10T22:33:39.360 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.360 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.361 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.361 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "key_file": { , 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.362 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.362 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:55.056 INFO:teuthology.orchestra.run.smithi092.stdout: "roption2": { , 2023-05-10T10:17:55.057 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "roption2", 2023-05-10T22:33:39.362 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.076 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2023-05-10T10:17:55.076 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.077 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.363 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.096 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:55.096 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:55.097 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.363 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster_level", , 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.364 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.364 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { , 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", 2023-05-10T22:33:39.365 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.365 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", 2023-05-10T22:33:39.365 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.366 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.366 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "discovery_interval", , 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.367 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.367 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.291 INFO:teuthology.orchestra.run.smithi092.stdout: "status", , 2023-05-10T10:17:55.291 INFO:teuthology.orchestra.run.smithi092.stdout: "telemetry", , 2023-05-10T10:17:55.292 INFO:teuthology.orchestra.run.smithi092.stdout: "volumes" 2023-05-10T22:33:39.367 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.191631036, , 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000074474 , 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.368 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, , 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, , 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2023-05-10T22:33:39.368 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.502 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { , 2023-05-10T10:17:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, , 2023-05-10T10:17:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-05-10T22:33:39.369 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, , 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, , 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-05-10T22:33:39.369 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.542 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 478365, , 2023-05-10T10:17:55.543 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, , 2023-05-10T10:17:55.543 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-05-10T22:33:39.370 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 259024, , 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 51569210, , 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-05-10T22:33:39.370 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000000000, , 2023-05-10T10:17:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { , 2023-05-10T10:17:55.731 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2023-05-10T22:33:39.371 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, , 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, , 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 148823, 2023-05-10T22:33:39.373 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.910+0000 7feeec3a6640 1 -- 172.21.15.92:0/2862986098 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee40b1700 con 0x7feee4146a20 , 2023-05-10T10:17:55.910+0000 7feedaffd640 1 -- 172.21.15.92:0/2862986098 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7feedc00abd0 con 0x7feee4146a20 , 2023-05-10T10:17:55.910+0000 7feedaffd640 1 --2- 172.21.15.92:0/2862986098 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7feec8066280 0x7feec8068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.374 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:56.475 INFO:tasks.ceph.ceph_manager.ceph:clean! , 2023-05-10T10:17:56.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-05-10T10:17:56.642+0000 7f4d17412640 1 Processor -- start 2023-05-10T22:33:39.375 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:57.022+0000 7f8644952640 1 --2- 172.21.15.92:0/1910604831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86400bc060 0x7f864013f360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86300097c0 tx=0x7f863001c3c0 comp rx=0 tx=0).stop , 2023-05-10T10:17:57.022+0000 7f8644952640 1 -- 172.21.15.92:0/1910604831 shutdown_connections , 2023-05-10T10:17:57.022+0000 7f8644952640 1 --2- 172.21.15.92:0/1910604831 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f86400b3310 0x7f86400b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.375 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fcbe8004d60 0x7fcc000adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc001de330 0x7fcc00142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc00142b70 0x7fcc000acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.376 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:58.846+0000 7f7e83fff640 1 -- 172.21.15.92:0/4079243528 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e900b67a0 con 0x7f7e9c0892e0 , 2023-05-10T10:17:58.846+0000 7f7e83fff640 1 -- 172.21.15.92:0/4079243528 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900b6b50 con 0x7f7e9c0892e0 , 2023-05-10T10:17:58.846+0000 7f7ea1ed1640 1 -- 172.21.15.92:0/4079243528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e900084a0 con 0x7f7e9c0892e0 2023-05-10T22:33:39.377 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.194+0000 7f0674339640 1 --2- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6816/2447806576,v1:172.21.15.92:6817/2447806576] conn(0x7f065006ec00 0x7f0650071050 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:59.194+0000 7f0674339640 1 -- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f06500662b0 msgr2=0x7f0650068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:17:59.194+0000 7f0674339640 1 --2- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f06500662b0 0x7f0650068760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f066c1308b0 tx=0x7f0668008040 comp rx=0 tx=0).stop 2023-05-10T22:33:39.377 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.510+0000 7fab6df60640 1 -- 172.21.15.92:0/779489884 shutdown_connections , 2023-05-10T10:17:59.510+0000 7fab6df60640 1 -- 172.21.15.92:0/779489884 wait complete. , 2023-05-10T10:17:59.533 INFO:tasks.ceph:Scrubbing osd.2 2023-05-10T22:33:39.378 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740a58c0 con 0x7fd7840bbc60 , 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7740a5a20 con 0x7fd7840bbc60 , 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740a5db0 con 0x7fd7840bbc60 2023-05-10T22:33:39.378 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 -- 172.21.15.92:0/640647466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48180ba9b0 msgr2=0x7f48180b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 --2- 172.21.15.92:0/640647466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48180ba9b0 0x7f48180b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 -- 172.21.15.92:0/640647466 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810009a70 con 0x7f48180ba5e0 2023-05-10T22:33:39.379 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.382+0000 7fc607fff640 1 -- 172.21.15.92:0/3699793562 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7fc6100eb160 con 0x7fc6200ac150 , 2023-05-10T10:18:00.530+0000 7fc62621f640 1 -- 172.21.15.92:0/3699793562 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc62004dd50 con 0x7fc60c066280 , 2023-05-10T10:18:00.530+0000 7fc607fff640 1 -- 172.21.15.92:0/3699793562 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc62004dd50 con 0x7fc60c066280 2023-05-10T22:33:39.379 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.878+0000 7f87a8699640 1 Processor -- start , 2023-05-10T10:18:00.878+0000 7f87a8699640 1 -- start start , 2023-05-10T10:18:00.878+0000 7f87a8699640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f87a00b1ec0 0x7f87a013bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.380 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8904142aa0 0x7f8904145520 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f890413c2f0 0x7f8904142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> v2:172.21.15.194:3300/0 conn(0x7f89040b1ec0 0x7f890413bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.380 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.402+0000 7fae42d18640 1 --2- 172.21.15.92:0/190621659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae3c146e30 0x7fae3c1e0220 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c410 tx=0x7fae3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:18:01.406+0000 7fae32ffd640 1 -- 172.21.15.92:0/190621659 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae340074e0 con 0x7fae3c146e30 , 2023-05-10T10:18:01.406+0000 7fae32ffd640 1 -- 172.21.15.92:0/190621659 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34007640 con 0x7fae3c146e30 2023-05-10T22:33:39.381 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 -- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1b90066280 msgr2=0x7f1b90068730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 --2- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1b90066280 0x7f1b90068730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005680 tx=0x7f1ba0003dd0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 -- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb0145100 msgr2=0x7f1bb00b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.381 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a5950 con 0x7f3bec142960 , 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be40a5ab0 con 0x7f3bec142960 , 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a5e40 con 0x7f3bec142960 2023-05-10T22:33:39.382 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.406+0000 7fbafce28640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbaf80b1ec0 0x7fbaf80b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:44774/0 (socket says 172.21.15.92:44774) , 2023-05-10T10:18:02.406+0000 7fbafce28640 1 -- 172.21.15.92:0/4219600763 learned_addr learned my addr 172.21.15.92:0/4219600763 (peer_addr_for_me v2:172.21.15.92:0/0) , 2023-05-10T10:18:02.406+0000 7fbaeffff640 1 --2- 172.21.15.92:0/4219600763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbaf8142560 0x7fbaf81e0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.383 WARNING:scrape:Ignoring malparsed backtrace: dumped all , 2023-05-10T10:18:02.894+0000 7f1236f98640 1 -- 172.21.15.92:0/4289600361 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f12140662b0 msgr2=0x7f1214068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:02.894+0000 7f1236f98640 1 --2- 172.21.15.92:0/4289600361 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f12140662b0 0x7f1214068760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1224007fd0 tx=0x7f1224009040 comp rx=0 tx=0).stop 2023-05-10T22:33:39.384 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.385 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:23.299 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:23.299 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:23.300 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.386 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:38.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-10T10:18:43.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-05-10T10:18:43.469+0000 7f59ae939640 1 Processor -- start 2023-05-10T22:33:39.387 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:43.679 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:43.679 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:43.680 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.388 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 --2- 172.21.15.92:0/1761086283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7f8004d10 0x7fb7f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 -- 172.21.15.92:0/1761086283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bb20 con 0x7fb7f80040a0 , 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 -- 172.21.15.92:0/1761086283 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bc60 con 0x7fb7f8004d10 2023-05-10T22:33:39.389 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:04.049 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:04.050 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:04.050 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.390 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:24.213+0000 7f01d088c640 1 -- start start , 2023-05-10T10:19:24.213+0000 7f01d088c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01cc0b40c0 0x7f01cc13cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:19:24.213+0000 7f01d088c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01cc0b5610 0x7f01cc13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.391 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:24.424 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:24.425 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:24.425 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.392 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:44.593+0000 7f3accff9640 1 -- 172.21.15.92:0/861657520 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7f3ab80b36f0 con 0x7f3ad00bc060 , 2023-05-10T10:19:44.737+0000 7f3ad58d1640 1 -- 172.21.15.92:0/861657520 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ad004dd50 con 0x7f3ab0066280 , 2023-05-10T10:19:44.745+0000 7f3accff9640 1 -- 172.21.15.92:0/861657520 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312088 (secure 0 0 0) 0x7f3ad004dd50 con 0x7f3ab0066280 2023-05-10T22:33:39.393 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:44.860 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:44.861 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:44.861 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.394 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:05.210 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:05.210 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:05.211 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.395 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:25.376+0000 7f3990a11640 1 --2- 172.21.15.92:0/3688155896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f398c13c5a0 0x7f398c14ca50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f398c0b19c0 tx=0x7f398401eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291d182945c6a452 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:20:25.376+0000 7f398affd640 1 -- 172.21.15.92:0/3688155896 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a7830 con 0x7f398c13c5a0 , 2023-05-10T10:20:25.376+0000 7f398affd640 1 -- 172.21.15.92:0/3688155896 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840a7990 con 0x7f398c13c5a0 2023-05-10T22:33:39.396 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.397 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:45.760+0000 7fbec659c640 1 --2- 172.21.15.92:0/1247818276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbec80b0be0 0x7fbec8144310 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800e870 tx=0x7fbeb800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:20:45.760+0000 7fbebf7fe640 1 -- 172.21.15.92:0/1247818276 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8009510 con 0x7fbec80b0be0 , 2023-05-10T10:20:45.760+0000 7fbebf7fe640 1 -- 172.21.15.92:0/1247818276 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb800eec0 con 0x7fbec80b0be0 2023-05-10T22:33:39.398 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:45.981 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:45.981 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:45.982 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.399 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:06.334 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:06.335 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:06.335 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.400 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:26.504+0000 7fde10f29640 1 -- 172.21.15.92:0/3502791245 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc0a7050 con 0x7fde0c0af9f0 , 2023-05-10T10:21:26.504+0000 7fde10f29640 1 -- 172.21.15.92:0/3502791245 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0ab980 con 0x7fde0c0af9f0 , 2023-05-10T10:21:26.504+0000 7fde1418f640 1 -- 172.21.15.92:0/3502791245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde0c0af9f0 msgr2=0x7fde0c149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.401 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.402 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:46.876+0000 7ff8ad7fa640 1 -- 172.21.15.92:0/1067990458 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800f040 con 0x7ff8b0146a20 , 2023-05-10T10:21:46.876+0000 7ff8b6d25640 1 -- 172.21.15.92:0/1067990458 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b006f2a0 con 0x7ff8b0146a20 , 2023-05-10T10:21:46.876+0000 7ff8ad7fa640 1 -- 172.21.15.92:0/1067990458 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800de80 con 0x7ff8b0146a20 2023-05-10T22:33:39.403 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:47.089 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:47.089 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:47.090 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.404 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:07.407+0000 7fa723b25640 1 --2- 172.21.15.92:0/1033317209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa71c0b29e0 0x7fa71c1d6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:22:07.407+0000 7fa723b25640 1 --2- 172.21.15.92:0/1033317209 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa708006f20 0x7fa71c1450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:22:07.407+0000 7fa723b25640 1 -- 172.21.15.92:0/1033317209 >> 172.21.15.92:0/1033317209 conn(0x7fa71c0b73c0 msgr2=0x7fa71c0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-10T22:33:39.405 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4003530 con 0x7fa6cc13c1d0 , 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986438495 0 0) 0x7fa6b4003530 con 0x7fa6cc13c1d0 , 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 >> v2:172.21.15.194:3300/0 conn(0x7fa6cc0afdf0 msgr2=0x7fa6cc1456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.406 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:27.815 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:27.815 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:27.816 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.407 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:47.999+0000 7f9e0b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c13b710 0x7f9e0c1e0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48960/0 (socket says 172.21.15.92:48960) , 2023-05-10T10:22:47.999+0000 7f9e0b7fe640 1 -- 172.21.15.92:0/4248290134 learned_addr learned my addr 172.21.15.92:0/4248290134 (peer_addr_for_me v2:172.21.15.92:0/0) , 2023-05-10T10:22:47.999+0000 7f9e0affd640 1 --2- 172.21.15.92:0/4248290134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e0c0b0250 0x7f9e0c12e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.408 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.409 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 --2- 172.21.15.92:0/2013621284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f115413d6d0 0x7f11541df230 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f114800f5a0 tx=0x7f114800f690 comp rx=0 tx=0).stop , 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 -- 172.21.15.92:0/2013621284 shutdown_connections , 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 --2- 172.21.15.92:0/2013621284 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f11541df770 0x7f11541dfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.410 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:28.734+0000 7f7fe5ffb640 1 -- 172.21.15.92:0/3273409605 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706832390 0 0) 0x7f7fe80b8d40 con 0x7f7fe8142860 , 2023-05-10T10:23:28.734+0000 7f7fe5ffb640 1 -- 172.21.15.92:0/3273409605 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe80b8ad0 con 0x7f7fe8142860 , 2023-05-10T10:23:28.734+0000 7f7fe67fc640 1 -- 172.21.15.92:0/3273409605 >> v1:172.21.15.194:6789/0 conn(0x7f7fe80b83a0 legacy=0x7f7fe806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:28.941 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:23:28.942 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:23:28.942 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:49.269+0000 7f3e09ffb640 1 -- 172.21.15.92:0/2342637346 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+313789 (secure 0 0 0) 0x7f3e1404dd50 con 0x7f3df80662b0 , 2023-05-10T10:23:49.269 INFO:teuthology.orchestra.run.smithi092.stdout: , dumped all 2023-05-10T22:33:39.413 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:09.465+0000 7fa5360ef640 1 --2- 172.21.15.92:0/1384444669 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa53013c5a0 0x7fa53014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:24:09.465+0000 7fa5348ec640 1 -- 172.21.15.92:0/1384444669 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514500000 0 0) 0x7fa530071ac0 con 0x7fa53013c1d0 , 2023-05-10T10:24:09.465+0000 7fa5348ec640 1 -- 172.21.15.92:0/1384444669 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c003530 con 0x7fa53013c1d0 2023-05-10T22:33:39.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.415 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 -- 172.21.15.92:0/3004979523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c00146a20 msgr2=0x7f6c001e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 --2- 172.21.15.92:0/3004979523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c00146a20 0x7f6c001e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 -- 172.21.15.92:0/3004979523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0009420 con 0x7f6c0014bac0 2023-05-10T22:33:39.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:30.045 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:30.045 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:30.046 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.419 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af006bed0 con 0x7f5af00b40c0 , 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af006bd40 con 0x7f5af00b40c0 , 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af00b5c80 con 0x7f5af00b40c0 2023-05-10T22:33:39.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:30.926+0000 7fed25012640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fed200b1c30 0x7fed200b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:25:30.926+0000 7fed25012640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed2013f400 0x7fed2013f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:25:30.926+0000 7fed25012640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed200b38c0 con 0x7fed200b0fb0 2023-05-10T22:33:39.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.422 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:51.297+0000 7f44f37fe640 1 -- 172.21.15.92:0/1921343202 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f44ec083d80 con 0x7f44fc0b0be0 , 2023-05-10T10:25:51.297+0000 7f44f1d9b640 1 --2- 172.21.15.92:0/1921343202 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f44dc0662b0 0x7f44dc068760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:25:51.297+0000 7f44f1d9b640 1 --2- 172.21.15.92:0/1921343202 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f44dc0662b0 0x7f44dc068760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44e40af000 tx=0x7f44e40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.423 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:11.652+0000 7f38bffff640 1 --2- 172.21.15.92:0/2226374949 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f38c00bc060 0x7f38c013f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:26:11.652+0000 7f38beffd640 1 -- 172.21.15.92:0/2226374949 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430380296 0 0) 0x7f38c008a060 con 0x7f38c013f8a0 , 2023-05-10T10:26:11.652+0000 7f38beffd640 1 -- 172.21.15.92:0/2226374949 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a8003530 con 0x7f38c013f8a0 2023-05-10T22:33:39.424 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.425 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:32.020+0000 7f2826ffd640 1 -- 172.21.15.92:0/3939449709 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820004020 con 0x7f283006ed00 , 2023-05-10T10:26:32.020+0000 7f2838360640 1 -- 172.21.15.92:0/3939449709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28300b2150 con 0x7f283006ed00 , 2023-05-10T10:26:32.020+0000 7f2826ffd640 1 -- 172.21.15.92:0/3939449709 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28200043b0 con 0x7f283006ed00 2023-05-10T22:33:39.426 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:52.379+0000 7f19997f0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1994145dd0 con 0x7f199413c5a0 , 2023-05-10T10:26:52.379+0000 7f19997f0640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19940b3c10 con 0x7f199413c1d0 , 2023-05-10T10:26:52.379+0000 7f19937fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f199413c5a0 0x7f199414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.427 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:52.580 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:52.580 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:52.581 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.428 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:12.742+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a480b2150 con 0x7f3a4813cba0 , 2023-05-10T10:27:12.742+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a480ae990 con 0x7f3a4813cba0 , 2023-05-10T10:27:12.746+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4804dd50 con 0x7f3a4813cba0 2023-05-10T22:33:39.430 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:33.094+0000 7f390659c640 1 --2- 172.21.15.92:0/2968357658 >> v2:172.21.15.194:3300/0 conn(0x7f39080afdf0 0x7f39081456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:27:33.094+0000 7f390759e640 1 --2- 172.21.15.92:0/2968357658 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f390813c5a0 0x7f39080ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:27:33.094+0000 7f3906d9d640 1 -- 172.21.15.92:0/2968357658 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f390813c5a0 msgr2=0x7f39080ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.430 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:33.298 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:27:33.299 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:27:33.299 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.432 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 -- 172.21.15.92:0/3384285672 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7f7de80067e0 con 0x7f7de00039b0 , 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 --2- 172.21.15.92:0/3384285672 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f7de4066280 0x7f7de4068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 -- 172.21.15.92:0/3384285672 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f7de811e330 con 0x7f7de00039b0 2023-05-10T22:33:39.433 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:13.825+0000 7fb39898c640 1 -- 172.21.15.92:0/4118702217 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39406e570 con 0x7fb39413c1d0 , 2023-05-10T10:28:13.829+0000 7fb39898c640 1 --2- 172.21.15.92:0/4118702217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb39413c1d0 0x7fb3940b1d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb3940b13c0 tx=0x7fb39001eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c990e17de74807ab server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:28:13.829+0000 7fb38b7fe640 1 -- 172.21.15.92:0/4118702217 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900a79d0 con 0x7fb39413c1d0 2023-05-10T22:33:39.434 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:14.032 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:14.032 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:14.033 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.435 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:34.381 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:34.381 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:34.382 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.436 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58940052d0 con 0x7f58a00b07d0 , 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894005430 con 0x7f58a00b07d0 , 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894012870 con 0x7f58a00b07d0 2023-05-10T22:33:39.438 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 -- 172.21.15.92:0/851029519 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187886880 0 0) 0x7f53b0003530 con 0x7f53c413c1d0 , 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 -- 172.21.15.92:0/851029519 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f53c413c5a0 msgr2=0x7f53c414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 --2- 172.21.15.92:0/851029519 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f53c413c5a0 0x7f53c414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.438 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:15.120 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:29:15.120 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:29:15.121 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.439 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 -- 172.21.15.92:0/3083124976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4dd00b5610 msgr2=0x7f4dd0139a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 --2- 172.21.15.92:0/3083124976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4dd00b5610 0x7f4dd0139a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0019150 tx=0x7f4dc001ad80 comp rx=0 tx=0).stop , 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 -- 172.21.15.92:0/3083124976 shutdown_connections 2023-05-10T22:33:39.441 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43c0afdf0 0x7fa43c142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43c0afdf0 0x7fa43c142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42862/0 (socket says 172.21.15.92:42862) , 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 -- 172.21.15.92:0/1418296414 learned_addr learned my addr 172.21.15.92:0/1418296414 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-05-10T22:33:39.442 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f3c0b40a0 0x7f9f3c147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f3c0b40a0 0x7f9f3c147240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47766/0 (socket says 172.21.15.92:47766) , 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 -- 172.21.15.92:0/1565947473 learned_addr learned my addr 172.21.15.92:0/1565947473 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-05-10T22:33:39.443 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.444 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 --2- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f02940662b0 0x7f0294068760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f02a80b2920 tx=0x7f029c008040 comp rx=0 tx=0).stop , 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 -- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0290004d10 msgr2=0x7f02a8143380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 --2- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0290004d10 0x7f02a8143380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f029800f810 tx=0x7f029800fcd0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.445 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:56.721+0000 7f0c20c0e640 1 -- 172.21.15.92:0/450291203 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c18005a60 con 0x7f0c08003b10 , 2023-05-10T10:30:56.721+0000 7f0c20c0e640 1 -- 172.21.15.92:0/450291203 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18005e50 con 0x7f0c08003b10 , 2023-05-10T10:30:56.725+0000 7f0c22c12640 1 -- 172.21.15.92:0/450291203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c08003b10 msgr2=0x7f0c08003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.447 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:56.938 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:56.939 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2023-05-10T10:31:12.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-10T22:33:39.448 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:17.288 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:17.288 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:17.289 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.449 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.450 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:57.811+0000 7f190d2d5640 1 --2- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18f4003b10 0x7f1908145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:31:57.811+0000 7f190dad6640 1 --2- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f19080b51a0 0x7f19080b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:31:57.811+0000 7f190cad4640 1 -- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f19080b51a0 msgr2=0x7f19080b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.451 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:18.167+0000 7f09f4855640 1 --2- 172.21.15.92:0/1532289199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b2ab0 0x7f09f01420d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09d800a960 tx=0x7f09d801d7c0 comp rx=0 tx=0).stop , 2023-05-10T10:32:18.167+0000 7f09f4855640 1 -- 172.21.15.92:0/1532289199 shutdown_connections , 2023-05-10T10:32:18.167+0000 7f09f4855640 1 --2- 172.21.15.92:0/1532289199 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f09f013b710 0x7f09f014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.453 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:38.515+0000 7fecf54f9640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006e7b0 con 0x7fecf00b0be0 , 2023-05-10T10:32:38.515+0000 7fecee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecf0146a20 0x7fecf0146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:32:38.515+0000 7fecee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecf0146a20 0x7fecf0146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53002/0 (socket says 172.21.15.92:53002) 2023-05-10T22:33:39.454 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:38.718 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:38.718 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:38.719 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.456 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:59.071 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:59.072 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:59.072 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.456 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:19.418 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:33:19.418 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:33:19.419 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.457 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 >> 172.21.15.92:0/1999965888 conn(0x7fe6600b73c0 msgr2=0x7fe6600b1c40 unknown :-1 s=STATE_NONE l=0).mark_down , 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 shutdown_connections , 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 wait complete. 2023-05-10T22:33:39.459 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:59.937+0000 7fee48ff9640 1 -- 172.21.15.92:0/1818378791 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44001210 con 0x7fee4c150b50 , 2023-05-10T10:33:59.937+0000 7fee50a12640 1 -- 172.21.15.92:0/1818378791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c151550 con 0x7fee4c150b50 , 2023-05-10T10:33:59.941+0000 7fee50a12640 1 -- 172.21.15.92:0/1818378791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c151550 con 0x7fee4c150b50 2023-05-10T22:33:39.460 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:34:20.288+0000 7f5664ff9640 1 -- 172.21.15.92:0/3694622742 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660007640 con 0x7f566814bac0 , 2023-05-10T10:34:20.288+0000 7f5664ff9640 1 -- 172.21.15.92:0/3694622742 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660004a30 con 0x7f566814bac0 , 2023-05-10T10:34:20.288+0000 7f566db6d640 1 -- 172.21.15.92:0/3694622742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56680b1700 con 0x7f566814bac0 2023-05-10T22:33:39.462 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:34:40.647+0000 7f8468ff9640 1 -- 172.21.15.92:0/2348478847 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8460005430 con 0x7f846c0af620 , 2023-05-10T10:34:40.647+0000 7f8468ff9640 1 -- 172.21.15.92:0/2348478847 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460003820 con 0x7f846c0af620 , 2023-05-10T10:34:40.647+0000 7f8472841640 1 -- 172.21.15.92:0/2348478847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c149fe0 con 0x7f846c0af620 2023-05-10T22:33:39.463 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:00.998+0000 7fad51708640 1 -- 172.21.15.92:0/1839056521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48009a70 con 0x7fad4c13c1d0 , 2023-05-10T10:35:00.998+0000 7fad5270a640 1 --2- 172.21.15.92:0/1839056521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad4c13c5a0 0x7fad4c1e58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! , 2023-05-10T10:35:00.998+0000 7fad51708640 1 --2- 172.21.15.92:0/1839056521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad4c13c1d0 0x7fad4c1e53a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e840 tx=0x7fad3c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.464 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:21.354+0000 7f39b5117640 1 --2- 172.21.15.92:0/3488893550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b013c5a0 0x7f39b0134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:35:21.354+0000 7f39b5117640 1 -- 172.21.15.92:0/3488893550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac009a70 con 0x7f39b00afdf0 , 2023-05-10T10:35:21.354+0000 7f39b4916640 1 --2- 172.21.15.92:0/3488893550 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f39b013c1d0 0x7f39b014c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-10T22:33:39.466 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:41.705+0000 7ff56146d640 1 --2- 172.21.15.92:0/3276553125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff55c1494a0 0x7ff55c1ddac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:35:41.705+0000 7ff56146d640 1 -- 172.21.15.92:0/3276553125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c009420 con 0x7ff55c0abda0 , 2023-05-10T10:35:41.705+0000 7ff56146d640 1 --2- 172.21.15.92:0/3276553125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c0abda0 0x7ff55c144ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff55800e930 tx=0x7ff55800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.467 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:02.052+0000 7fca8b798640 1 -- 172.21.15.92:0/1107612113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8406f2a0 con 0x7fca840b0be0 , 2023-05-10T10:36:02.052+0000 7fca8b798640 1 -- 172.21.15.92:0/1107612113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca840b1700 con 0x7fca840b0be0 , 2023-05-10T10:36:02.056+0000 7fca79ffb640 1 -- 172.21.15.92:0/1107612113 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7fca800073d0 con 0x7fca840b0be0 2023-05-10T22:33:39.469 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:22.412+0000 7fd9a6ffd640 1 -- 172.21.15.92:0/1005505257 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7fd9a00eb540 con 0x7fd9b013c1d0 , 2023-05-10T10:36:22.560+0000 7fd9b8325640 1 -- 172.21.15.92:0/1005505257 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9b013f380 con 0x7fd994066280 , 2023-05-10T10:36:22.568+0000 7fd9a6ffd640 1 -- 172.21.15.92:0/1005505257 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+317459 (secure 0 0 0) 0x7fd9ac005080 con 0x7fd994066280 2023-05-10T22:33:39.470 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 --2- 172.21.15.92:0/1529654815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f9c071c40 0x7f9f9c06f5b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800c490 tx=0x7f9f9800c950 comp rx=0 tx=0).stop , 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 -- 172.21.15.92:0/1529654815 shutdown_connections , 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 --2- 172.21.15.92:0/1529654815 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f9f800662b0 0x7f9f80068760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.472 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 >> 172.21.15.92:0/2541361664 conn(0x7f170406ba10 msgr2=0x7f17040b6a30 unknown :-1 s=STATE_NONE l=0).mark_down , 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 shutdown_connections , 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 wait complete. 2023-05-10T22:33:39.474 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:37:23.647 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:37:23.648 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:37:23.648 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.475 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:04.145+0000 7f3127462640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f31200b0fb0 0x7f31200b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:38:04.145+0000 7f3127462640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31200b38c0 con 0x7f31200b1c30 , 2023-05-10T10:38:04.145+0000 7f3127462640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312006df80 con 0x7f312013f400 2023-05-10T22:33:39.476 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:24.492+0000 7f5ba4b69640 1 --2- 172.21.15.92:0/3454878715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba00afdf0 0x7f5ba01456c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5b900097c0 tx=0x7f5b9001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19728d4d796bdca8 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:38:24.492+0000 7f5b97fff640 1 -- 172.21.15.92:0/3454878715 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b900ac640 con 0x7f5ba00afdf0 , 2023-05-10T10:38:24.492+0000 7f5b97fff640 1 -- 172.21.15.92:0/3454878715 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b900ac7a0 con 0x7f5ba00afdf0 2023-05-10T22:33:39.478 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:44.844+0000 7f3d49812640 1 -- 172.21.15.92:0/2590519533 wait complete. , 2023-05-10T10:38:44.844+0000 7f3d49812640 1 Processor -- start , 2023-05-10T10:38:44.844+0000 7f3d49812640 1 -- start start 2023-05-10T22:33:39.479 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:05.195+0000 7f8b8e59c640 1 --2- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b9813c1d0 0x7f8b98143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:39:05.195+0000 7f8b9759e640 1 --2- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8b9813c5a0 0x7f8b98143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:39:05.195+0000 7f8b96d9d640 1 -- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8b9813c5a0 msgr2=0x7f8b98143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.481 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:25.554+0000 7f109ffff640 1 -- 172.21.15.92:0/1930888513 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7f10900baba0 con 0x7f10a813c1d0 , 2023-05-10T10:39:25.554+0000 7f109ffff640 1 --2- 172.21.15.92:0/1930888513 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1088066280 0x7f1088068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:39:25.554+0000 7f109ffff640 1 -- 172.21.15.92:0/1930888513 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f109011f130 con 0x7f10a813c1d0 2023-05-10T22:33:39.483 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7faafc066280 0x7faafc068730 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab18146a20 0x7fab18070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fab180b0be0 0x7fab180715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.484 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:40:26.572+0000 7f84cc2fc640 1 -- start start , 2023-05-10T10:40:26.576+0000 7f84cc2fc640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f84c413c1d0 0x7f84c40b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:40:26.576+0000 7f84cc2fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f84c40afdf0 0x7f84c41456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.485 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:40:46.923+0000 7ff0b1336640 1 -- 172.21.15.92:0/998423516 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267002128 0 0) 0x7ff0ac0b17f0 con 0x7ff0ac0af620 , 2023-05-10T10:40:46.923+0000 7ff0b2b39640 1 --2- 172.21.15.92:0/998423516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ac071c40 0x7ff0ac149c80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac073770 tx=0x7ff0a401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d3eb06948a13b9 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:40:46.923+0000 7ff0b1336640 1 -- 172.21.15.92:0/998423516 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40ab600 con 0x7ff0ac071c40 2023-05-10T22:33:39.487 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:07.270+0000 7f5622525640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b16f0 con 0x7f561c0b3310 , 2023-05-10T10:41:07.270+0000 7f561bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0aee10 0x7f561c13dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:41:07.270+0000 7f561bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0aee10 0x7f561c13dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33418/0 (socket says 172.21.15.92:33418) 2023-05-10T22:33:39.488 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:27.613+0000 7f822b539640 1 -- 172.21.15.92:0/2379905546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822414d6d0 con 0x7f8224071c40 , 2023-05-10T10:41:27.613+0000 7f8219ffb640 1 -- 172.21.15.92:0/2379905546 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000ce30 con 0x7f8224071c40 , 2023-05-10T10:41:27.613+0000 7f822b539640 1 -- 172.21.15.92:0/2379905546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822414d540 con 0x7f8224071c40 2023-05-10T22:33:39.490 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:48.116+0000 7fa70de41640 1 -- 172.21.15.92:0/246480377 shutdown_connections , 2023-05-10T10:41:48.116+0000 7fa70de41640 1 --2- 172.21.15.92:0/246480377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa70813c5a0 0x7fa70813b090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:41:48.116+0000 7fa70de41640 1 --2- 172.21.15.92:0/246480377 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa70813c1d0 0x7fa708138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.492 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:42:28.678+0000 7f3ba71a2640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba006df80 con 0x7f3ba00b0fb0 , 2023-05-10T10:42:28.678+0000 7f3ba71a2640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba006e260 con 0x7f3ba00b1c30 , 2023-05-10T10:42:28.678+0000 7f3ba69a1640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3ba013f400 0x7f3ba014c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57762/0 (socket says 172.21.15.92:57762) 2023-05-10T22:33:39.493 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:42:49.017+0000 7f080a59c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f080c0b0be0 0x7f080c13d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49914/0 (socket says 172.21.15.92:49914) , 2023-05-10T10:42:49.017+0000 7f080a59c640 1 -- 172.21.15.92:0/2078512857 learned_addr learned my addr 172.21.15.92:0/2078512857 (peer_addr_for_me v1:172.21.15.92:0/0) , 2023-05-10T10:42:49.017+0000 7f0809d9b640 1 -- 172.21.15.92:0/2078512857 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767686239 0 0) 0x7f080c13d8d0 con 0x7f080c0b0be0 2023-05-10T22:33:39.495 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:09.357+0000 7fc6f96d6640 1 --2- 172.21.15.92:0/4265691674 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc6f413c5a0 0x7fc6f414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:43:09.361+0000 7fc6f8ed5640 1 --2- 172.21.15.92:0/4265691674 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc6f413c5a0 0x7fc6f414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:09.361+0000 7fc6f8ed5640 1 -- 172.21.15.92:0/4265691674 >> v1:172.21.15.92:6789/0 conn(0x7fc6f40afdf0 legacy=0x7fc6f41456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.496 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:29.700+0000 7fe73d569640 1 --2- 172.21.15.92:0/1350475399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe724003980 0x7fe724003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:29.700+0000 7fe73d569640 1 -- 172.21.15.92:0/1350475399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724003530 con 0x7fe7240047d0 , 2023-05-10T10:43:29.700+0000 7fe73e56b640 1 --2- 172.21.15.92:0/1350475399 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe724005380 0x7fe72400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-10T22:33:39.497 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728142650 con 0x7f97280bc060 , 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728139c10 con 0x7f97280b3310 , 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972813b830 con 0x7f972813f8a0 2023-05-10T22:33:39.498 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:50.419+0000 7f3e2bfff640 1 -- 172.21.15.92:0/1064163243 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f3e24084240 con 0x7f3e340b1ec0 , 2023-05-10T10:43:50.419+0000 7f3e2a7fc640 1 --2- 172.21.15.92:0/1064163243 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f3e14066280 0x7f3e14068730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:43:50.419+0000 7f3e2a7fc640 1 --2- 172.21.15.92:0/1064163243 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f3e14066280 0x7f3e14068730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c006fd0 tx=0x7f3e1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.498 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.083+0000 7f47cd18f640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f47c813c1d0 0x7f47c80b1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:52234/0 (socket says 172.21.15.92:52234) , 2023-05-10T10:43:51.083+0000 7f47cd18f640 1 -- 172.21.15.92:0/3437986477 learned_addr learned my addr 172.21.15.92:0/3437986477 (peer_addr_for_me v1:172.21.15.92:0/0) , 2023-05-10T10:43:51.083+0000 7f47cc98e640 1 --2- 172.21.15.92:0/3437986477 >> v2:172.21.15.92:3300/0 conn(0x7f47c80afdf0 0x7f47c81456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.499 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.427+0000 7fe239a95640 1 -- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe2340b0250 msgr2=0x7fe2200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:43:51.427+0000 7fe239a95640 1 --2- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe2340b0250 0x7fe2200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:51.427+0000 7fe239a95640 1 -- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe220004d10 msgr2=0x7fe220005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.500 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 -- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff398146e30 msgr2=0x7ff3981e1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 --2- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff398146e30 0x7ff3981e1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 -- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3980b1300 msgr2=0x7ff3981e0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.500 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fad68145600 0x7fad681e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad680b7a20 0x7fad681e0a40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7fad54066280 0x7fad54068730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.501 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 -- 172.21.15.92:0/3107303740 >> v2:172.21.15.194:3300/0 conn(0x7f0e580b8140 msgr2=0x7f0e580b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 --2- 172.21.15.92:0/3107303740 >> v2:172.21.15.194:3300/0 conn(0x7f0e580b8140 0x7f0e580b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 -- 172.21.15.92:0/3107303740 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58148cd0 con 0x7f0e580ba5e0 2023-05-10T22:33:39.501 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:44:20.439 INFO:tasks.ceph.osd.6:Stopped , 2023-05-10T10:44:20.440 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2023-05-10T10:44:20.440 INFO:teuthology.orchestra.run:waiting for 300 2023-05-10T22:33:39.501 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== , 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:*hv01.front.sepi 44.4.53.2 2 u 13 64 377 0.066 -3.000 2.285 , 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 64 377 0.089 -0.413 3.395 2023-05-10T22:33:39.546 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-3879-ge28f4bf0 (e28f4bf0df5d53a504fe40e32a78dd63506e2421) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fccef9616e1] , 2: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7fccef961895] 2023-05-10T22:33:39.548 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:51.314+0000 7f2418a12640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24141455a0 con 0x7f24140b1c30 , 2023-05-10T10:17:51.314+0000 7f24137fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140b0fb0 0x7f241406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:17:51.314+0000 7f24137fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140b0fb0 0x7f241406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:36526/0 (socket says 172.21.15.92:36526) 2023-05-10T22:33:39.549 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 -- 172.21.15.92:0/4047004657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7aa01dc6f0 msgr2=0x7f7aa01dcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 --2- 172.21.15.92:0/4047004657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7aa01dc6f0 0x7f7aa01dcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:51.642+0000 7f7a9e59c640 1 -- 172.21.15.92:0/4047004657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa00b7dc0 con 0x7f7a8c004d60 2023-05-10T22:33:39.550 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c08146010 0x7f2c08141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c0813f490 0x7f2c08140fb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:52.122+0000 7f2c0f28c640 1 --2- 172.21.15.92:0/1260993314 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c080b3700 0x7f2c08143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.551 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 -- start start , 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cd80b1300 0x7f9cd81468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:52.594+0000 7f9cdcaf2640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f9cd8146e30 0x7f9cd814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.552 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 -- start start , 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f88cc069af0 0x7f88cc1398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:52.914+0000 7f88d27ef640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88cc0b8450 0x7f88cc1ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.553 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.129 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster": { , 2023-05-10T10:17:53.130 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:53.130 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", 2023-05-10T22:33:39.554 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.149 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.150 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.150 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.555 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "0", , 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "6", , 2023-05-10T10:17:53.169 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.556 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.189 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.557 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "how frequently to wake up and attempt optimization", , 2023-05-10T10:17:53.209 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:53.229 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.559 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.248 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.249 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.249 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Prometheus container image", 2023-05-10T22:33:39.559 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.269 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.560 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: "daemon_cache_timeout": { , 2023-05-10T10:17:53.289 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "daemon_cache_timeout", 2023-05-10T22:33:39.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.309 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "*", , 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.329 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.348 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.349 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "customized SSH config file to connect to managed hosts", , 2023-05-10T10:17:53.368 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.387 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.388 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.407 INFO:teuthology.orchestra.run.smithi092.stdout: "ALERTMANAGER_API_HOST": { , 2023-05-10T10:17:53.408 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "ALERTMANAGER_API_HOST", , 2023-05-10T10:17:53.408 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.427 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", 2023-05-10T22:33:39.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.446 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.565 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.465 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:53.465 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.466 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.565 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.485 INFO:teuthology.orchestra.run.smithi092.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-05-10T22:33:39.566 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.505 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_ADMIN_RESOURCE", , 2023-05-10T10:17:53.505 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.506 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.566 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.525 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.525 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:53.526 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.567 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: ], , 2023-05-10T10:17:53.545 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", 2023-05-10T22:33:39.567 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.564 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.567 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.583 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:53.583 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.584 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.568 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:53.603 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.568 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.623 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.569 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.642 INFO:teuthology.orchestra.run.smithi092.stdout: "interval": { 2023-05-10T22:33:39.569 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.661 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "port", , 2023-05-10T10:17:53.662 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2023-05-10T10:17:53.662 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.570 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:53.681 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.570 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.701 INFO:teuthology.orchestra.run.smithi092.stdout: ], , 2023-05-10T10:17:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", 2023-05-10T22:33:39.571 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:53.721 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", 2023-05-10T22:33:39.571 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.740 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:53.741 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.571 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.760 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.779 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:53.780 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:53.780 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.573 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.818 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:53.819 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:53.819 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.573 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.838 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "3.0", 2023-05-10T22:33:39.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { , 2023-05-10T10:17:53.858 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", 2023-05-10T22:33:39.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.877 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:53.877 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.878 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.575 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.897 INFO:teuthology.orchestra.run.smithi092.stdout: "server_port": { 2023-05-10T22:33:39.576 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:53.917 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster_level": { 2023-05-10T22:33:39.577 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:53.936 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.578 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.955 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.955 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:53.956 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.579 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "roption2": { , 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "roption2", , 2023-05-10T10:17:53.975 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.580 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:53.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:53.995 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.581 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.014 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.581 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.033 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.034 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.582 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.053 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.582 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:54.072 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.073 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.582 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.583 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:54.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.583 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-05-10T22:33:39.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [ 2023-05-10T22:33:39.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.171 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2023-05-10T10:17:54.171 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.172 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.585 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.191 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.585 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_cluster": { , 2023-05-10T10:17:54.210 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", 2023-05-10T22:33:39.586 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.229 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Password to authenticate with", , 2023-05-10T10:17:54.230 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.230 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.586 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "0", , 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "0", , 2023-05-10T10:17:54.249 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "6", 2023-05-10T22:33:39.587 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.269 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.587 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.288 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-05-10T22:33:39.588 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.307 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.308 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", 2023-05-10T22:33:39.588 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.327 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], 2023-05-10T22:33:39.589 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Jaeger query container image", , 2023-05-10T10:17:54.346 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.347 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.589 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.366 INFO:teuthology.orchestra.run.smithi092.stdout: "daemon_cache_timeout": { 2023-05-10T22:33:39.590 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", , 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.590 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.406 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.406 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "*", , 2023-05-10T10:17:54.407 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.591 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "admin", , 2023-05-10T10:17:54.426 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.592 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.446 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-05-10T22:33:39.593 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.465 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.465 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.466 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.594 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: "ALERTMANAGER_API_HOST": { , 2023-05-10T10:17:54.485 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "ALERTMANAGER_API_HOST", 2023-05-10T22:33:39.594 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.504 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.504 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.505 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.595 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.524 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.597 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.544 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", 2023-05-10T22:33:39.597 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.598 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.582 INFO:teuthology.orchestra.run.smithi092.stdout: "RGW_API_ADMIN_RESOURCE": { , 2023-05-10T10:17:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_ADMIN_RESOURCE", , 2023-05-10T10:17:54.583 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.599 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:54.603 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.600 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.623 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.601 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.642 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.643 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.601 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.662 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2023-05-10T22:33:39.602 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:54.682 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.602 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.702 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.603 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.603 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.741 INFO:teuthology.orchestra.run.smithi092.stdout: "port": { , 2023-05-10T10:17:54.742 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "port", , 2023-05-10T10:17:54.742 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", 2023-05-10T22:33:39.604 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:54.761 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", 2023-05-10T22:33:39.604 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.780 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.780 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" , 2023-05-10T10:17:54.781 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2023-05-10T22:33:39.604 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2023-05-10T22:33:39.605 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:54.820 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.605 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster_level", , 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:54.839 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.606 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.858 INFO:teuthology.orchestra.run.smithi092.stdout: "debug", , 2023-05-10T10:17:54.859 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:54.859 INFO:teuthology.orchestra.run.smithi092.stdout: "info", 2023-05-10T22:33:39.606 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.878 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.607 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.897 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2023-05-10T10:17:54.897 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:54.898 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", 2023-05-10T22:33:39.607 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "float", , 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:54.917 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.607 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:54.937 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { 2023-05-10T22:33:39.608 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster", , 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", , 2023-05-10T10:17:54.956 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.608 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.975 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.609 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2023-05-10T10:17:54.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.609 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "key_file": { , 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2023-05-10T10:17:55.016 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", 2023-05-10T22:33:39.609 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2023-05-10T10:17:55.037 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", 2023-05-10T22:33:39.610 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:55.056 INFO:teuthology.orchestra.run.smithi092.stdout: "roption2": { , 2023-05-10T10:17:55.057 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "roption2", 2023-05-10T22:33:39.610 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.076 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2023-05-10T10:17:55.076 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.077 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.611 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.096 INFO:teuthology.orchestra.run.smithi092.stdout: "error", , 2023-05-10T10:17:55.096 INFO:teuthology.orchestra.run.smithi092.stdout: "info", , 2023-05-10T10:17:55.097 INFO:teuthology.orchestra.run.smithi092.stdout: "warning" 2023-05-10T22:33:39.611 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_cluster_level", , 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:55.116 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.612 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:55.135 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:55.136 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.612 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": { , 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_to_file", , 2023-05-10T10:17:55.155 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "bool", 2023-05-10T22:33:39.613 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.174 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2023-05-10T10:17:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2023-05-10T10:17:55.175 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] 2023-05-10T22:33:39.613 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2023-05-10T10:17:55.194 INFO:teuthology.orchestra.run.smithi092.stdout: "log_level": { , 2023-05-10T10:17:55.195 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "log_level", 2023-05-10T22:33:39.613 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2023-05-10T10:17:55.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, 2023-05-10T22:33:39.614 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.233 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2023-05-10T10:17:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2023-05-10T10:17:55.234 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", 2023-05-10T22:33:39.614 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "discovery_interval", , 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2023-05-10T10:17:55.253 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", 2023-05-10T22:33:39.615 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "False", , 2023-05-10T10:17:55.272 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", 2023-05-10T22:33:39.615 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.291 INFO:teuthology.orchestra.run.smithi092.stdout: "status", , 2023-05-10T10:17:55.291 INFO:teuthology.orchestra.run.smithi092.stdout: "telemetry", , 2023-05-10T10:17:55.292 INFO:teuthology.orchestra.run.smithi092.stdout: "volumes" 2023-05-10T22:33:39.615 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.191631036, , 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000074474 , 2023-05-10T10:17:55.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-05-10T22:33:39.616 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, , 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, , 2023-05-10T10:17:55.483 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2023-05-10T22:33:39.616 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.502 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { , 2023-05-10T10:17:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, , 2023-05-10T10:17:55.503 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-05-10T22:33:39.617 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, , 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, , 2023-05-10T10:17:55.522 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-05-10T22:33:39.617 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.542 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 478365, , 2023-05-10T10:17:55.543 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, , 2023-05-10T10:17:55.543 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-05-10T22:33:39.618 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 259024, , 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 51569210, , 2023-05-10T10:17:55.562 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-05-10T22:33:39.618 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000000000, , 2023-05-10T10:17:55.730 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { , 2023-05-10T10:17:55.731 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2023-05-10T22:33:39.618 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, , 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, , 2023-05-10T10:17:55.750 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 148823, 2023-05-10T22:33:39.621 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:55.910+0000 7feeec3a6640 1 -- 172.21.15.92:0/2862986098 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee40b1700 con 0x7feee4146a20 , 2023-05-10T10:17:55.910+0000 7feedaffd640 1 -- 172.21.15.92:0/2862986098 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7feedc00abd0 con 0x7feee4146a20 , 2023-05-10T10:17:55.910+0000 7feedaffd640 1 --2- 172.21.15.92:0/2862986098 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7feec8066280 0x7feec8068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.622 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:56.475 INFO:tasks.ceph.ceph_manager.ceph:clean! , 2023-05-10T10:17:56.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-05-10T10:17:56.642+0000 7f4d17412640 1 Processor -- start 2023-05-10T22:33:39.623 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:57.022+0000 7f8644952640 1 --2- 172.21.15.92:0/1910604831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86400bc060 0x7f864013f360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86300097c0 tx=0x7f863001c3c0 comp rx=0 tx=0).stop , 2023-05-10T10:17:57.022+0000 7f8644952640 1 -- 172.21.15.92:0/1910604831 shutdown_connections , 2023-05-10T10:17:57.022+0000 7f8644952640 1 --2- 172.21.15.92:0/1910604831 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f86400b3310 0x7f86400b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.624 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fcbe8004d60 0x7fcc000adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc001de330 0x7fcc00142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:17:58.374+0000 7fcc06c41640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcc00142b70 0x7fcc000acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.624 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:58.846+0000 7f7e83fff640 1 -- 172.21.15.92:0/4079243528 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e900b67a0 con 0x7f7e9c0892e0 , 2023-05-10T10:17:58.846+0000 7f7e83fff640 1 -- 172.21.15.92:0/4079243528 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900b6b50 con 0x7f7e9c0892e0 , 2023-05-10T10:17:58.846+0000 7f7ea1ed1640 1 -- 172.21.15.92:0/4079243528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e900084a0 con 0x7f7e9c0892e0 2023-05-10T22:33:39.625 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.194+0000 7f0674339640 1 --2- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6816/2447806576,v1:172.21.15.92:6817/2447806576] conn(0x7f065006ec00 0x7f0650071050 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:17:59.194+0000 7f0674339640 1 -- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f06500662b0 msgr2=0x7f0650068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:17:59.194+0000 7f0674339640 1 --2- 172.21.15.92:0/873660968 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f06500662b0 0x7f0650068760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f066c1308b0 tx=0x7f0668008040 comp rx=0 tx=0).stop 2023-05-10T22:33:39.625 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.510+0000 7fab6df60640 1 -- 172.21.15.92:0/779489884 shutdown_connections , 2023-05-10T10:17:59.510+0000 7fab6df60640 1 -- 172.21.15.92:0/779489884 wait complete. , 2023-05-10T10:17:59.533 INFO:tasks.ceph:Scrubbing osd.2 2023-05-10T22:33:39.626 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740a58c0 con 0x7fd7840bbc60 , 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7740a5a20 con 0x7fd7840bbc60 , 2023-05-10T10:17:59.862+0000 7fd7827fc640 1 -- 172.21.15.92:0/2238465090 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740a5db0 con 0x7fd7840bbc60 2023-05-10T22:33:39.626 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 -- 172.21.15.92:0/640647466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48180ba9b0 msgr2=0x7f48180b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 --2- 172.21.15.92:0/640647466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48180ba9b0 0x7f48180b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:00.198+0000 7f481d8c0640 1 -- 172.21.15.92:0/640647466 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810009a70 con 0x7f48180ba5e0 2023-05-10T22:33:39.627 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.382+0000 7fc607fff640 1 -- 172.21.15.92:0/3699793562 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7fc6100eb160 con 0x7fc6200ac150 , 2023-05-10T10:18:00.530+0000 7fc62621f640 1 -- 172.21.15.92:0/3699793562 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc62004dd50 con 0x7fc60c066280 , 2023-05-10T10:18:00.530+0000 7fc607fff640 1 -- 172.21.15.92:0/3699793562 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc62004dd50 con 0x7fc60c066280 2023-05-10T22:33:39.627 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:00.878+0000 7f87a8699640 1 Processor -- start , 2023-05-10T10:18:00.878+0000 7f87a8699640 1 -- start start , 2023-05-10T10:18:00.878+0000 7f87a8699640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f87a00b1ec0 0x7f87a013bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.628 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8904142aa0 0x7f8904145520 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f890413c2f0 0x7f8904142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.222+0000 7f89097ce640 1 --2- 172.21.15.92:0/4215554610 >> v2:172.21.15.194:3300/0 conn(0x7f89040b1ec0 0x7f890413bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.629 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.402+0000 7fae42d18640 1 --2- 172.21.15.92:0/190621659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae3c146e30 0x7fae3c1e0220 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae3400c410 tx=0x7fae3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:18:01.406+0000 7fae32ffd640 1 -- 172.21.15.92:0/190621659 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae340074e0 con 0x7fae3c146e30 , 2023-05-10T10:18:01.406+0000 7fae32ffd640 1 -- 172.21.15.92:0/190621659 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34007640 con 0x7fae3c146e30 2023-05-10T22:33:39.629 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 -- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1b90066280 msgr2=0x7f1b90068730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 --2- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1b90066280 0x7f1b90068730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005680 tx=0x7f1ba0003dd0 comp rx=0 tx=0).stop , 2023-05-10T10:18:01.746+0000 7f1bb4b52640 1 -- 172.21.15.92:0/2878629315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb0145100 msgr2=0x7f1bb00b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.630 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a5950 con 0x7f3bec142960 , 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be40a5ab0 con 0x7f3bec142960 , 2023-05-10T10:18:02.226+0000 7f3bf0a0e640 1 -- 172.21.15.92:0/2816273331 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a5e40 con 0x7f3bec142960 2023-05-10T22:33:39.630 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.406+0000 7fbafce28640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbaf80b1ec0 0x7fbaf80b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:44774/0 (socket says 172.21.15.92:44774) , 2023-05-10T10:18:02.406+0000 7fbafce28640 1 -- 172.21.15.92:0/4219600763 learned_addr learned my addr 172.21.15.92:0/4219600763 (peer_addr_for_me v2:172.21.15.92:0/0) , 2023-05-10T10:18:02.406+0000 7fbaeffff640 1 --2- 172.21.15.92:0/4219600763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbaf8142560 0x7fbaf81e0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.632 WARNING:scrape:Ignoring malparsed backtrace: dumped all , 2023-05-10T10:18:02.894+0000 7f1236f98640 1 -- 172.21.15.92:0/4289600361 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f12140662b0 msgr2=0x7f1214068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:18:02.894+0000 7f1236f98640 1 --2- 172.21.15.92:0/4289600361 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f12140662b0 0x7f1214068760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1224007fd0 tx=0x7f1224009040 comp rx=0 tx=0).stop 2023-05-10T22:33:39.633 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:02.953 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.634 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:23.299 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:23.299 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:23.300 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.635 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:38.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-05-10T10:18:43.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-05-10T10:18:43.469+0000 7f59ae939640 1 Processor -- start 2023-05-10T22:33:39.635 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:18:43.679 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:43.679 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:18:43.680 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.637 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 --2- 172.21.15.92:0/1761086283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7f8004d10 0x7fb7f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 -- 172.21.15.92:0/1761086283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bb20 con 0x7fb7f80040a0 , 2023-05-10T10:19:03.841+0000 7fb80a7fc640 1 -- 172.21.15.92:0/1761086283 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bc60 con 0x7fb7f8004d10 2023-05-10T22:33:39.637 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:04.049 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:04.050 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:04.050 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.639 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:24.213+0000 7f01d088c640 1 -- start start , 2023-05-10T10:19:24.213+0000 7f01d088c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01cc0b40c0 0x7f01cc13cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:19:24.213+0000 7f01d088c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01cc0b5610 0x7f01cc13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.639 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:24.424 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:24.425 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:24.425 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.640 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:44.593+0000 7f3accff9640 1 -- 172.21.15.92:0/861657520 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7f3ab80b36f0 con 0x7f3ad00bc060 , 2023-05-10T10:19:44.737+0000 7f3ad58d1640 1 -- 172.21.15.92:0/861657520 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ad004dd50 con 0x7f3ab0066280 , 2023-05-10T10:19:44.745+0000 7f3accff9640 1 -- 172.21.15.92:0/861657520 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312088 (secure 0 0 0) 0x7f3ad004dd50 con 0x7f3ab0066280 2023-05-10T22:33:39.642 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:19:44.860 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:44.861 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:19:44.861 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.642 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:05.210 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:05.210 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:05.211 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.644 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:25.376+0000 7f3990a11640 1 --2- 172.21.15.92:0/3688155896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f398c13c5a0 0x7f398c14ca50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f398c0b19c0 tx=0x7f398401eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291d182945c6a452 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:20:25.376+0000 7f398affd640 1 -- 172.21.15.92:0/3688155896 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39840a7830 con 0x7f398c13c5a0 , 2023-05-10T10:20:25.376+0000 7f398affd640 1 -- 172.21.15.92:0/3688155896 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840a7990 con 0x7f398c13c5a0 2023-05-10T22:33:39.644 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:25.593 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.645 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:45.760+0000 7fbec659c640 1 --2- 172.21.15.92:0/1247818276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbec80b0be0 0x7fbec8144310 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800e870 tx=0x7fbeb800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:20:45.760+0000 7fbebf7fe640 1 -- 172.21.15.92:0/1247818276 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8009510 con 0x7fbec80b0be0 , 2023-05-10T10:20:45.760+0000 7fbebf7fe640 1 -- 172.21.15.92:0/1247818276 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb800eec0 con 0x7fbec80b0be0 2023-05-10T22:33:39.647 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:20:45.981 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:45.981 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:20:45.982 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.647 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:06.334 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:06.335 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:06.335 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.649 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:26.504+0000 7fde10f29640 1 -- 172.21.15.92:0/3502791245 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc0a7050 con 0x7fde0c0af9f0 , 2023-05-10T10:21:26.504+0000 7fde10f29640 1 -- 172.21.15.92:0/3502791245 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0ab980 con 0x7fde0c0af9f0 , 2023-05-10T10:21:26.504+0000 7fde1418f640 1 -- 172.21.15.92:0/3502791245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde0c0af9f0 msgr2=0x7fde0c149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.649 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:26.714 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.650 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:46.876+0000 7ff8ad7fa640 1 -- 172.21.15.92:0/1067990458 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800f040 con 0x7ff8b0146a20 , 2023-05-10T10:21:46.876+0000 7ff8b6d25640 1 -- 172.21.15.92:0/1067990458 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b006f2a0 con 0x7ff8b0146a20 , 2023-05-10T10:21:46.876+0000 7ff8ad7fa640 1 -- 172.21.15.92:0/1067990458 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800de80 con 0x7ff8b0146a20 2023-05-10T22:33:39.651 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:21:47.089 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:47.089 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:21:47.090 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.652 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:07.407+0000 7fa723b25640 1 --2- 172.21.15.92:0/1033317209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa71c0b29e0 0x7fa71c1d6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:22:07.407+0000 7fa723b25640 1 --2- 172.21.15.92:0/1033317209 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa708006f20 0x7fa71c1450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:22:07.407+0000 7fa723b25640 1 -- 172.21.15.92:0/1033317209 >> 172.21.15.92:0/1033317209 conn(0x7fa71c0b73c0 msgr2=0x7fa71c0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-05-10T22:33:39.654 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4003530 con 0x7fa6cc13c1d0 , 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986438495 0 0) 0x7fa6b4003530 con 0x7fa6cc13c1d0 , 2023-05-10T10:22:27.607+0000 7fa6c37fe640 1 -- 172.21.15.92:0/1348070574 >> v2:172.21.15.194:3300/0 conn(0x7fa6cc0afdf0 msgr2=0x7fa6cc1456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.654 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:27.815 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:27.815 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:27.816 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.655 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:47.999+0000 7f9e0b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e0c13b710 0x7f9e0c1e0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:48960/0 (socket says 172.21.15.92:48960) , 2023-05-10T10:22:47.999+0000 7f9e0b7fe640 1 -- 172.21.15.92:0/4248290134 learned_addr learned my addr 172.21.15.92:0/4248290134 (peer_addr_for_me v2:172.21.15.92:0/0) , 2023-05-10T10:22:47.999+0000 7f9e0affd640 1 --2- 172.21.15.92:0/4248290134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9e0c0b0250 0x7f9e0c12e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.656 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:22:48.206 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.657 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 --2- 172.21.15.92:0/2013621284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f115413d6d0 0x7f11541df230 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f114800f5a0 tx=0x7f114800f690 comp rx=0 tx=0).stop , 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 -- 172.21.15.92:0/2013621284 shutdown_connections , 2023-05-10T10:23:08.534+0000 7f115a8b6640 1 --2- 172.21.15.92:0/2013621284 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f11541df770 0x7f11541dfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.659 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:28.734+0000 7f7fe5ffb640 1 -- 172.21.15.92:0/3273409605 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706832390 0 0) 0x7f7fe80b8d40 con 0x7f7fe8142860 , 2023-05-10T10:23:28.734+0000 7f7fe5ffb640 1 -- 172.21.15.92:0/3273409605 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe80b8ad0 con 0x7f7fe8142860 , 2023-05-10T10:23:28.734+0000 7f7fe67fc640 1 -- 172.21.15.92:0/3273409605 >> v1:172.21.15.194:6789/0 conn(0x7f7fe80b83a0 legacy=0x7f7fe806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.659 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:28.941 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:23:28.942 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:23:28.942 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.661 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:23:49.269+0000 7f3e09ffb640 1 -- 172.21.15.92:0/2342637346 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+313789 (secure 0 0 0) 0x7f3e1404dd50 con 0x7f3df80662b0 , 2023-05-10T10:23:49.269 INFO:teuthology.orchestra.run.smithi092.stdout: , dumped all 2023-05-10T22:33:39.662 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:09.465+0000 7fa5360ef640 1 --2- 172.21.15.92:0/1384444669 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa53013c5a0 0x7fa53014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:24:09.465+0000 7fa5348ec640 1 -- 172.21.15.92:0/1384444669 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514500000 0 0) 0x7fa530071ac0 con 0x7fa53013c1d0 , 2023-05-10T10:24:09.465+0000 7fa5348ec640 1 -- 172.21.15.92:0/1384444669 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c003530 con 0x7fa53013c1d0 2023-05-10T22:33:39.662 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:09.665 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.664 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 -- 172.21.15.92:0/3004979523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c00146a20 msgr2=0x7f6c001e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 --2- 172.21.15.92:0/3004979523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c00146a20 0x7f6c001e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:24:29.836+0000 7f6bff7fe640 1 -- 172.21.15.92:0/3004979523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0009420 con 0x7f6c0014bac0 2023-05-10T22:33:39.665 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:30.045 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:30.045 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:30.046 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.666 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:24:50.403 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.667 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af006bed0 con 0x7f5af00b40c0 , 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af006bd40 con 0x7f5af00b40c0 , 2023-05-10T10:25:10.575+0000 7f5af7b8f640 1 -- 172.21.15.92:0/3364453750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af00b5c80 con 0x7f5af00b40c0 2023-05-10T22:33:39.668 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:30.926+0000 7fed25012640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fed200b1c30 0x7fed200b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:25:30.926+0000 7fed25012640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed2013f400 0x7fed2013f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:25:30.926+0000 7fed25012640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed200b38c0 con 0x7fed200b0fb0 2023-05-10T22:33:39.669 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:25:31.128 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.670 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:25:51.297+0000 7f44f37fe640 1 -- 172.21.15.92:0/1921343202 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f44ec083d80 con 0x7f44fc0b0be0 , 2023-05-10T10:25:51.297+0000 7f44f1d9b640 1 --2- 172.21.15.92:0/1921343202 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f44dc0662b0 0x7f44dc068760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:25:51.297+0000 7f44f1d9b640 1 --2- 172.21.15.92:0/1921343202 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f44dc0662b0 0x7f44dc068760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44e40af000 tx=0x7f44e40a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.671 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:11.652+0000 7f38bffff640 1 --2- 172.21.15.92:0/2226374949 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f38c00bc060 0x7f38c013f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:26:11.652+0000 7f38beffd640 1 -- 172.21.15.92:0/2226374949 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430380296 0 0) 0x7f38c008a060 con 0x7f38c013f8a0 , 2023-05-10T10:26:11.652+0000 7f38beffd640 1 -- 172.21.15.92:0/2226374949 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a8003530 con 0x7f38c013f8a0 2023-05-10T22:33:39.672 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:11.858 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.673 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:32.020+0000 7f2826ffd640 1 -- 172.21.15.92:0/3939449709 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820004020 con 0x7f283006ed00 , 2023-05-10T10:26:32.020+0000 7f2838360640 1 -- 172.21.15.92:0/3939449709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28300b2150 con 0x7f283006ed00 , 2023-05-10T10:26:32.020+0000 7f2826ffd640 1 -- 172.21.15.92:0/3939449709 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28200043b0 con 0x7f283006ed00 2023-05-10T22:33:39.675 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:52.379+0000 7f19997f0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1994145dd0 con 0x7f199413c5a0 , 2023-05-10T10:26:52.379+0000 7f19997f0640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19940b3c10 con 0x7f199413c1d0 , 2023-05-10T10:26:52.379+0000 7f19937fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f199413c5a0 0x7f199414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.675 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:26:52.580 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:52.580 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:26:52.581 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.677 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:12.742+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a480b2150 con 0x7f3a4813cba0 , 2023-05-10T10:27:12.742+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a480ae990 con 0x7f3a4813cba0 , 2023-05-10T10:27:12.746+0000 7f3a4fcdf640 1 -- 172.21.15.92:0/3965005968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4804dd50 con 0x7f3a4813cba0 2023-05-10T22:33:39.678 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:33.094+0000 7f390659c640 1 --2- 172.21.15.92:0/2968357658 >> v2:172.21.15.194:3300/0 conn(0x7f39080afdf0 0x7f39081456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:27:33.094+0000 7f390759e640 1 --2- 172.21.15.92:0/2968357658 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f390813c5a0 0x7f39080ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:27:33.094+0000 7f3906d9d640 1 -- 172.21.15.92:0/2968357658 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f390813c5a0 msgr2=0x7f39080ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.678 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:33.298 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:27:33.299 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:27:33.299 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.680 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 -- 172.21.15.92:0/3384285672 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7f7de80067e0 con 0x7f7de00039b0 , 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 --2- 172.21.15.92:0/3384285672 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f7de4066280 0x7f7de4068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:27:53.465+0000 7f7ddffff640 1 -- 172.21.15.92:0/3384285672 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f7de811e330 con 0x7f7de00039b0 2023-05-10T22:33:39.681 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:13.825+0000 7fb39898c640 1 -- 172.21.15.92:0/4118702217 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39406e570 con 0x7fb39413c1d0 , 2023-05-10T10:28:13.829+0000 7fb39898c640 1 --2- 172.21.15.92:0/4118702217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb39413c1d0 0x7fb3940b1d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb3940b13c0 tx=0x7fb39001eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c990e17de74807ab server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:28:13.829+0000 7fb38b7fe640 1 -- 172.21.15.92:0/4118702217 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3900a79d0 con 0x7fb39413c1d0 2023-05-10T22:33:39.683 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:14.032 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:14.032 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:14.033 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.683 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:34.381 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:34.381 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:28:34.382 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.685 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58940052d0 con 0x7f58a00b07d0 , 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894005430 con 0x7f58a00b07d0 , 2023-05-10T10:28:54.559+0000 7f588effd640 1 -- 172.21.15.92:0/3081025262 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894012870 con 0x7f58a00b07d0 2023-05-10T22:33:39.686 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 -- 172.21.15.92:0/851029519 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187886880 0 0) 0x7f53b0003530 con 0x7f53c413c1d0 , 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 -- 172.21.15.92:0/851029519 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f53c413c5a0 msgr2=0x7f53c414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:29:14.919+0000 7f53c27fc640 1 --2- 172.21.15.92:0/851029519 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f53c413c5a0 0x7f53c414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.687 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:15.120 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:29:15.120 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:29:15.121 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.689 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 -- 172.21.15.92:0/3083124976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4dd00b5610 msgr2=0x7f4dd0139a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 --2- 172.21.15.92:0/3083124976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4dd00b5610 0x7f4dd0139a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0019150 tx=0x7f4dc001ad80 comp rx=0 tx=0).stop , 2023-05-10T10:29:35.458+0000 7f4dd7b25640 1 -- 172.21.15.92:0/3083124976 shutdown_connections 2023-05-10T22:33:39.690 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43c0afdf0 0x7fa43c142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa43c0afdf0 0x7fa43c142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42862/0 (socket says 172.21.15.92:42862) , 2023-05-10T10:29:55.650+0000 7fa43b7fe640 1 -- 172.21.15.92:0/1418296414 learned_addr learned my addr 172.21.15.92:0/1418296414 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-05-10T22:33:39.692 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f3c0b40a0 0x7f9f3c147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f3c0b40a0 0x7f9f3c147240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47766/0 (socket says 172.21.15.92:47766) , 2023-05-10T10:30:16.002+0000 7f9f3affd640 1 -- 172.21.15.92:0/1565947473 learned_addr learned my addr 172.21.15.92:0/1565947473 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-05-10T22:33:39.692 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:16.208 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.694 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 --2- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f02940662b0 0x7f0294068760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f02a80b2920 tx=0x7f029c008040 comp rx=0 tx=0).stop , 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 -- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0290004d10 msgr2=0x7f02a8143380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:30:36.537+0000 7f02ad6f2640 1 --2- 172.21.15.92:0/375532541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0290004d10 0x7f02a8143380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f029800f810 tx=0x7f029800fcd0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.695 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:56.721+0000 7f0c20c0e640 1 -- 172.21.15.92:0/450291203 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c18005a60 con 0x7f0c08003b10 , 2023-05-10T10:30:56.721+0000 7f0c20c0e640 1 -- 172.21.15.92:0/450291203 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18005e50 con 0x7f0c08003b10 , 2023-05-10T10:30:56.725+0000 7f0c22c12640 1 -- 172.21.15.92:0/450291203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c08003b10 msgr2=0x7f0c08003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.697 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:30:56.938 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:30:56.939 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2023-05-10T10:31:12.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-05-10T22:33:39.698 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:17.288 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:17.288 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:17.289 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.699 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:31:37.635 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.700 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:31:57.811+0000 7f190d2d5640 1 --2- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18f4003b10 0x7f1908145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:31:57.811+0000 7f190dad6640 1 --2- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f19080b51a0 0x7f19080b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:31:57.811+0000 7f190cad4640 1 -- 172.21.15.92:0/4082919707 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f19080b51a0 msgr2=0x7f19080b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.701 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:18.167+0000 7f09f4855640 1 --2- 172.21.15.92:0/1532289199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09f00b2ab0 0x7f09f01420d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f09d800a960 tx=0x7f09d801d7c0 comp rx=0 tx=0).stop , 2023-05-10T10:32:18.167+0000 7f09f4855640 1 -- 172.21.15.92:0/1532289199 shutdown_connections , 2023-05-10T10:32:18.167+0000 7f09f4855640 1 --2- 172.21.15.92:0/1532289199 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f09f013b710 0x7f09f014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.703 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:38.515+0000 7fecf54f9640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006e7b0 con 0x7fecf00b0be0 , 2023-05-10T10:32:38.515+0000 7fecee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecf0146a20 0x7fecf0146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:32:38.515+0000 7fecee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecf0146a20 0x7fecf0146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53002/0 (socket says 172.21.15.92:53002) 2023-05-10T22:33:39.704 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:38.718 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:38.718 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:38.719 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.706 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:32:59.071 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:59.072 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-05-10T10:11:58.869124+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=11, tm_sec=58, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:32:59.072 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-05-10T10:12:05.078151+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=5, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.706 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:19.418 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:33:19.418 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:33:19.419 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.708 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 >> 172.21.15.92:0/1999965888 conn(0x7fe6600b73c0 msgr2=0x7fe6600b1c40 unknown :-1 s=STATE_NONE l=0).mark_down , 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 shutdown_connections , 2023-05-10T10:33:39.750+0000 7fe6680e5640 1 -- 172.21.15.92:0/1999965888 wait complete. 2023-05-10T22:33:39.709 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:33:59.937+0000 7fee48ff9640 1 -- 172.21.15.92:0/1818378791 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44001210 con 0x7fee4c150b50 , 2023-05-10T10:33:59.937+0000 7fee50a12640 1 -- 172.21.15.92:0/1818378791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c151550 con 0x7fee4c150b50 , 2023-05-10T10:33:59.941+0000 7fee50a12640 1 -- 172.21.15.92:0/1818378791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c151550 con 0x7fee4c150b50 2023-05-10T22:33:39.711 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:34:20.288+0000 7f5664ff9640 1 -- 172.21.15.92:0/3694622742 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660007640 con 0x7f566814bac0 , 2023-05-10T10:34:20.288+0000 7f5664ff9640 1 -- 172.21.15.92:0/3694622742 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660004a30 con 0x7f566814bac0 , 2023-05-10T10:34:20.288+0000 7f566db6d640 1 -- 172.21.15.92:0/3694622742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56680b1700 con 0x7f566814bac0 2023-05-10T22:33:39.712 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:34:40.647+0000 7f8468ff9640 1 -- 172.21.15.92:0/2348478847 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8460005430 con 0x7f846c0af620 , 2023-05-10T10:34:40.647+0000 7f8468ff9640 1 -- 172.21.15.92:0/2348478847 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460003820 con 0x7f846c0af620 , 2023-05-10T10:34:40.647+0000 7f8472841640 1 -- 172.21.15.92:0/2348478847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c149fe0 con 0x7f846c0af620 2023-05-10T22:33:39.714 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:00.998+0000 7fad51708640 1 -- 172.21.15.92:0/1839056521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48009a70 con 0x7fad4c13c1d0 , 2023-05-10T10:35:00.998+0000 7fad5270a640 1 --2- 172.21.15.92:0/1839056521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad4c13c5a0 0x7fad4c1e58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! , 2023-05-10T10:35:00.998+0000 7fad51708640 1 --2- 172.21.15.92:0/1839056521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad4c13c1d0 0x7fad4c1e53a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e840 tx=0x7fad3c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.715 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:21.354+0000 7f39b5117640 1 --2- 172.21.15.92:0/3488893550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b013c5a0 0x7f39b0134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:35:21.354+0000 7f39b5117640 1 -- 172.21.15.92:0/3488893550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac009a70 con 0x7f39b00afdf0 , 2023-05-10T10:35:21.354+0000 7f39b4916640 1 --2- 172.21.15.92:0/3488893550 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f39b013c1d0 0x7f39b014c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-10T22:33:39.717 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:35:41.705+0000 7ff56146d640 1 --2- 172.21.15.92:0/3276553125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff55c1494a0 0x7ff55c1ddac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:35:41.705+0000 7ff56146d640 1 -- 172.21.15.92:0/3276553125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c009420 con 0x7ff55c0abda0 , 2023-05-10T10:35:41.705+0000 7ff56146d640 1 --2- 172.21.15.92:0/3276553125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c0abda0 0x7ff55c144ba0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff55800e930 tx=0x7ff55800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.718 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:02.052+0000 7fca8b798640 1 -- 172.21.15.92:0/1107612113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8406f2a0 con 0x7fca840b0be0 , 2023-05-10T10:36:02.052+0000 7fca8b798640 1 -- 172.21.15.92:0/1107612113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca840b1700 con 0x7fca840b0be0 , 2023-05-10T10:36:02.056+0000 7fca79ffb640 1 -- 172.21.15.92:0/1107612113 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7fca800073d0 con 0x7fca840b0be0 2023-05-10T22:33:39.719 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:22.412+0000 7fd9a6ffd640 1 -- 172.21.15.92:0/1005505257 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169682 (secure 0 0 0) 0x7fd9a00eb540 con 0x7fd9b013c1d0 , 2023-05-10T10:36:22.560+0000 7fd9b8325640 1 -- 172.21.15.92:0/1005505257 --> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9b013f380 con 0x7fd994066280 , 2023-05-10T10:36:22.568+0000 7fd9a6ffd640 1 -- 172.21.15.92:0/1005505257 <== mgr.4103 v2:172.21.15.92:6832/3754859973 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+317459 (secure 0 0 0) 0x7fd9ac005080 con 0x7fd994066280 2023-05-10T22:33:39.721 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 --2- 172.21.15.92:0/1529654815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f9c071c40 0x7f9f9c06f5b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800c490 tx=0x7f9f9800c950 comp rx=0 tx=0).stop , 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 -- 172.21.15.92:0/1529654815 shutdown_connections , 2023-05-10T10:36:42.923+0000 7f9fa382d640 1 --2- 172.21.15.92:0/1529654815 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f9f800662b0 0x7f9f80068760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.723 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 >> 172.21.15.92:0/2541361664 conn(0x7f170406ba10 msgr2=0x7f17040b6a30 unknown :-1 s=STATE_NONE l=0).mark_down , 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 shutdown_connections , 2023-05-10T10:37:03.270+0000 7f17093bc640 1 -- 172.21.15.92:0/2541361664 wait complete. 2023-05-10T22:33:39.725 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:37:23.647 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:37:23.648 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-05-10T10:12:09.108796+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=9, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) , 2023-05-10T10:37:23.648 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-05-10T10:12:10.115235+0000 time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=12, tm_sec=10, tm_wday=2, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=5, tm_mday=10, tm_hour=10, tm_min=17, tm_sec=57, tm_wday=2, tm_yday=130, tm_isdst=0) 2023-05-10T22:33:39.726 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:04.145+0000 7f3127462640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f31200b0fb0 0x7f31200b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:38:04.145+0000 7f3127462640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31200b38c0 con 0x7f31200b1c30 , 2023-05-10T10:38:04.145+0000 7f3127462640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312006df80 con 0x7f312013f400 2023-05-10T22:33:39.728 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:24.492+0000 7f5ba4b69640 1 --2- 172.21.15.92:0/3454878715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ba00afdf0 0x7f5ba01456c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5b900097c0 tx=0x7f5b9001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19728d4d796bdca8 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:38:24.492+0000 7f5b97fff640 1 -- 172.21.15.92:0/3454878715 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b900ac640 con 0x7f5ba00afdf0 , 2023-05-10T10:38:24.492+0000 7f5b97fff640 1 -- 172.21.15.92:0/3454878715 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b900ac7a0 con 0x7f5ba00afdf0 2023-05-10T22:33:39.729 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:38:44.844+0000 7f3d49812640 1 -- 172.21.15.92:0/2590519533 wait complete. , 2023-05-10T10:38:44.844+0000 7f3d49812640 1 Processor -- start , 2023-05-10T10:38:44.844+0000 7f3d49812640 1 -- start start 2023-05-10T22:33:39.731 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:05.195+0000 7f8b8e59c640 1 --2- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b9813c1d0 0x7f8b98143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:39:05.195+0000 7f8b9759e640 1 --2- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8b9813c5a0 0x7f8b98143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:39:05.195+0000 7f8b96d9d640 1 -- 172.21.15.92:0/575793892 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8b9813c5a0 msgr2=0x7f8b98143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.732 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:25.554+0000 7f109ffff640 1 -- 172.21.15.92:0/1930888513 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 85658+0+0 (secure 0 0 0) 0x7f10900baba0 con 0x7f10a813c1d0 , 2023-05-10T10:39:25.554+0000 7f109ffff640 1 --2- 172.21.15.92:0/1930888513 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f1088066280 0x7f1088068730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:39:25.554+0000 7f109ffff640 1 -- 172.21.15.92:0/1930888513 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f109011f130 con 0x7f10a813c1d0 2023-05-10T22:33:39.734 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7faafc066280 0x7faafc068730 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab18146a20 0x7fab18070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:39:46.057+0000 7fab2008c640 1 --2- 172.21.15.92:0/4028494466 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fab180b0be0 0x7fab180715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:40:26.572+0000 7f84cc2fc640 1 -- start start , 2023-05-10T10:40:26.576+0000 7f84cc2fc640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f84c413c1d0 0x7f84c40b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-05-10T10:40:26.576+0000 7f84cc2fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f84c40afdf0 0x7f84c41456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-05-10T22:33:39.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:40:46.923+0000 7ff0b1336640 1 -- 172.21.15.92:0/998423516 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267002128 0 0) 0x7ff0ac0b17f0 con 0x7ff0ac0af620 , 2023-05-10T10:40:46.923+0000 7ff0b2b39640 1 --2- 172.21.15.92:0/998423516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ac071c40 0x7ff0ac149c80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac073770 tx=0x7ff0a401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d3eb06948a13b9 server_cookie=0 in_seq=0 out_seq=0 , 2023-05-10T10:40:46.923+0000 7ff0b1336640 1 -- 172.21.15.92:0/998423516 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40ab600 con 0x7ff0ac071c40 2023-05-10T22:33:39.739 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:07.270+0000 7f5622525640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b16f0 con 0x7f561c0b3310 , 2023-05-10T10:41:07.270+0000 7f561bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0aee10 0x7f561c13dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:41:07.270+0000 7f561bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0aee10 0x7f561c13dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:33418/0 (socket says 172.21.15.92:33418) 2023-05-10T22:33:39.740 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:27.613+0000 7f822b539640 1 -- 172.21.15.92:0/2379905546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822414d6d0 con 0x7f8224071c40 , 2023-05-10T10:41:27.613+0000 7f8219ffb640 1 -- 172.21.15.92:0/2379905546 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000ce30 con 0x7f8224071c40 , 2023-05-10T10:41:27.613+0000 7f822b539640 1 -- 172.21.15.92:0/2379905546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822414d540 con 0x7f8224071c40 2023-05-10T22:33:39.742 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:41:48.116+0000 7fa70de41640 1 -- 172.21.15.92:0/246480377 shutdown_connections , 2023-05-10T10:41:48.116+0000 7fa70de41640 1 --2- 172.21.15.92:0/246480377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa70813c5a0 0x7fa70813b090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:41:48.116+0000 7fa70de41640 1 --2- 172.21.15.92:0/246480377 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa70813c1d0 0x7fa708138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.744 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:42:28.678+0000 7f3ba71a2640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba006df80 con 0x7f3ba00b0fb0 , 2023-05-10T10:42:28.678+0000 7f3ba71a2640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba006e260 con 0x7f3ba00b1c30 , 2023-05-10T10:42:28.678+0000 7f3ba69a1640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3ba013f400 0x7f3ba014c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:57762/0 (socket says 172.21.15.92:57762) 2023-05-10T22:33:39.745 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:42:49.017+0000 7f080a59c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f080c0b0be0 0x7f080c13d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49914/0 (socket says 172.21.15.92:49914) , 2023-05-10T10:42:49.017+0000 7f080a59c640 1 -- 172.21.15.92:0/2078512857 learned_addr learned my addr 172.21.15.92:0/2078512857 (peer_addr_for_me v1:172.21.15.92:0/0) , 2023-05-10T10:42:49.017+0000 7f0809d9b640 1 -- 172.21.15.92:0/2078512857 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767686239 0 0) 0x7f080c13d8d0 con 0x7f080c0b0be0 2023-05-10T22:33:39.747 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:09.357+0000 7fc6f96d6640 1 --2- 172.21.15.92:0/4265691674 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc6f413c5a0 0x7fc6f414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:43:09.361+0000 7fc6f8ed5640 1 --2- 172.21.15.92:0/4265691674 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc6f413c5a0 0x7fc6f414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:09.361+0000 7fc6f8ed5640 1 -- 172.21.15.92:0/4265691674 >> v1:172.21.15.92:6789/0 conn(0x7fc6f40afdf0 legacy=0x7fc6f41456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.748 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:29.700+0000 7fe73d569640 1 --2- 172.21.15.92:0/1350475399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe724003980 0x7fe724003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:29.700+0000 7fe73d569640 1 -- 172.21.15.92:0/1350475399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724003530 con 0x7fe7240047d0 , 2023-05-10T10:43:29.700+0000 7fe73e56b640 1 --2- 172.21.15.92:0/1350475399 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe724005380 0x7fe72400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-05-10T22:33:39.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728142650 con 0x7f97280bc060 , 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728139c10 con 0x7f97280b3310 , 2023-05-10T10:43:50.051+0000 7f972d725640 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972813b830 con 0x7f972813f8a0 2023-05-10T22:33:39.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:50.419+0000 7f3e2bfff640 1 -- 172.21.15.92:0/1064163243 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6163+0+0 (secure 0 0 0) 0x7f3e24084240 con 0x7f3e340b1ec0 , 2023-05-10T10:43:50.419+0000 7f3e2a7fc640 1 --2- 172.21.15.92:0/1064163243 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f3e14066280 0x7f3e14068730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-05-10T10:43:50.419+0000 7f3e2a7fc640 1 --2- 172.21.15.92:0/1064163243 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7f3e14066280 0x7f3e14068730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c006fd0 tx=0x7f3e1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-05-10T22:33:39.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.083+0000 7f47cd18f640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f47c813c1d0 0x7f47c80b1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:52234/0 (socket says 172.21.15.92:52234) , 2023-05-10T10:43:51.083+0000 7f47cd18f640 1 -- 172.21.15.92:0/3437986477 learned_addr learned my addr 172.21.15.92:0/3437986477 (peer_addr_for_me v1:172.21.15.92:0/0) , 2023-05-10T10:43:51.083+0000 7f47cc98e640 1 --2- 172.21.15.92:0/3437986477 >> v2:172.21.15.92:3300/0 conn(0x7f47c80afdf0 0x7f47c81456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-05-10T22:33:39.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.427+0000 7fe239a95640 1 -- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe2340b0250 msgr2=0x7fe2200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:43:51.427+0000 7fe239a95640 1 --2- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe2340b0250 0x7fe2200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:51.427+0000 7fe239a95640 1 -- 172.21.15.92:0/1513620299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe220004d10 msgr2=0x7fe220005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-05-10T22:33:39.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 -- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff398146e30 msgr2=0x7ff3981e1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 --2- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff398146e30 0x7ff3981e1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:51.755+0000 7ff39dac1640 1 -- 172.21.15.92:0/4190139927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3980b1300 msgr2=0x7ff3981e0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-05-10T22:33:39.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fad68145600 0x7fad681e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad680b7a20 0x7fad681e0a40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:53.603+0000 7fad6d4d7640 1 --2- 172.21.15.92:0/1704352750 >> [v2:172.21.15.92:6832/3754859973,v1:172.21.15.92:6833/3754859973] conn(0x7fad54066280 0x7fad54068730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-05-10T22:33:39.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 -- 172.21.15.92:0/3107303740 >> v2:172.21.15.194:3300/0 conn(0x7f0e580b8140 msgr2=0x7f0e580b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 --2- 172.21.15.92:0/3107303740 >> v2:172.21.15.194:3300/0 conn(0x7f0e580b8140 0x7f0e580b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-05-10T10:43:54.863+0000 7f0e5659c640 1 -- 172.21.15.92:0/3107303740 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58148cd0 con 0x7f0e580ba5e0 2023-05-10T22:33:39.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:44:20.439 INFO:tasks.ceph.osd.6:Stopped , 2023-05-10T10:44:20.440 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2023-05-10T10:44:20.440 INFO:teuthology.orchestra.run:waiting for 300 2023-05-10T22:33:39.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== , 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:*hv01.front.sepi 44.4.53.2 2 u 13 64 377 0.066 -3.000 2.285 , 2023-05-10T10:47:13.354 INFO:teuthology.orchestra.run.smithi092.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 64 377 0.089 -0.413 3.395 2023-05-10T22:33:42.455 WARNING:scrape:Misunderstood line: 2023-05-10T22:33:46.097 WARNING:scrape:Misunderstood line: 2023-05-10T22:34:39.767 INFO:scrape:Found 19 distinct failure reasons 2023-05-10T22:34:39.768 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-05-10T22:34:39.768 INFO:scrape:3 jobs: ['7268492', '7268413', '7268280'] 2023-05-10T22:34:39.769 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-05-10T22:34:39.769 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-05-10T22:34:39.769 INFO:scrape: 2023-05-10T22:34:39.770 INFO:scrape:Dead: b'2023-05-10T19:47:26.879 DEBUG:teuthology.exit:Finished running handlers' 2023-05-10T22:34:39.770 INFO:scrape:11 jobs: ['7268402', '7268418', '7268462', '7268453', '7268258', '7268323', '7268405', '7268369', '7268432', '7268307', '7268298'] 2023-05-10T22:34:39.771 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-05-10T22:34:39.771 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/postgres}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'wsync/no}', 'wsync/yes}'] 2023-05-10T22:34:39.771 INFO:scrape: 2023-05-10T22:34:39.772 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi092 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-05-10T22:34:39.772 INFO:scrape:1 jobs: ['7268495'] 2023-05-10T22:34:39.772 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-05-10T22:34:39.773 INFO:scrape: 2023-05-10T22:34:39.773 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi169 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-05-10T22:34:39.773 INFO:scrape:2 jobs: ['7268356', '7268415'] 2023-05-10T22:34:39.774 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'objectstore-ec/bluestore-comp', 'osd}'] 2023-05-10T22:34:39.774 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2023-05-10T22:34:39.774 INFO:scrape: 2023-05-10T22:34:39.775 INFO:scrape:Failure: Command failed on smithi100 with status 8: 'rm -f /tmp/linux-image.deb && echo linux-image-6.3.0-gd8cc0e0d9358_6.3.0-gd8cc0e0d9358-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testing/d8cc0e0d93583718825c83a107c1ec8808dfb2dc/ubuntu/focal/flavors/default/pool/main/l/linux-6.3.0-gd8cc0e0d9358/ --input-file=-' 2023-05-10T22:34:39.775 INFO:scrape:4 jobs: ['7268510', '7268454', '7268441', '7268506'] 2023-05-10T22:34:39.775 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_20.04}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2023-05-10T22:34:39.776 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/mds-flush}', 'tasks/metrics}', 'tasks/test_journal_migration}', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-05-10T22:34:39.776 INFO:scrape: 2023-05-10T22:34:39.776 INFO:scrape:Failure: Found coredumps on ubuntu@smithi118.front.sepia.ceph.com 2023-05-10T22:34:39.777 INFO:scrape:1 jobs: ['7268416'] 2023-05-10T22:34:39.777 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}'] 2023-05-10T22:34:39.777 INFO:scrape: 2023-05-10T22:34:39.778 INFO:scrape:Failure: "1683685660.2707245 mon.a (mon.0) 464 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-05-10T22:34:39.778 INFO:scrape:1 jobs: ['7268254'] 2023-05-10T22:34:39.778 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2023-05-10T22:34:39.778 INFO:scrape: 2023-05-10T22:34:39.779 INFO:scrape:Dead: b'2023-05-10T22:33:23.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-05-10T22:34:39.779 INFO:scrape:1 jobs: ['7268520'] 2023-05-10T22:34:39.779 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-05-10T22:34:39.779 INFO:scrape: 2023-05-10T22:34:39.780 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2023-05-10T22:34:39.780 INFO:scrape:1 jobs: ['7268376'] 2023-05-10T22:34:39.780 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-05-10T22:34:39.781 INFO:scrape: 2023-05-10T22:34:39.781 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-05-10T22:34:39.781 INFO:scrape:1 jobs: ['7268499'] 2023-05-10T22:34:39.782 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-05-10T22:34:39.782 INFO:scrape: 2023-05-10T22:34:39.782 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-05-10T22:34:39.782 INFO:scrape:2 jobs: ['7268468', '7268392'] 2023-05-10T22:34:39.783 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-10T22:34:39.783 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-10T22:34:39.784 INFO:scrape: 2023-05-10T22:34:39.784 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-05-10T22:34:39.784 INFO:scrape:1 jobs: ['7268472'] 2023-05-10T22:34:39.784 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-05-10T22:34:39.785 INFO:scrape: 2023-05-10T22:34:39.785 INFO:scrape:Failure: Command failed on smithi093 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-05-10T22:34:39.785 INFO:scrape:4 jobs: ['7268265', '7268465', '7268318', '7268433'] 2023-05-10T22:34:39.786 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-05-10T22:34:39.786 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-05-10T22:34:39.786 INFO:scrape: 2023-05-10T22:34:39.787 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/dbench.sh 2023-05-10T22:34:39.787 INFO:scrape:3 jobs: ['7268428', '7268302', '7268463'] 2023-05-10T22:34:39.787 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}'] 2023-05-10T22:34:39.787 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}'] 2023-05-10T22:34:39.788 INFO:scrape: 2023-05-10T22:34:39.788 INFO:scrape:Crash: Command failed (workunit test suites/ffsb.sh) on smithi121 with status 77: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh' 2023-05-10T22:34:39.788 INFO:scrape:ceph version 18.0.0-3879-ge28f4bf0 (e28f4bf0df5d53a504fe40e32a78dd63506e2421) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f1715c7b420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f171511e2b1] 5: /usr/lib/ceph/libceph-common.so.2(+0x31e413) [0x7f171511e413] 6: (Client::add_update_cap(Inode*, MetaSession*, unsigned long, unsigned int, unsigned int, unsigned int, unsigned int, inodeno_t, int, UserPerm const&)+0xc21) [0x5623150b3ab1] 7: (Client::add_update_inode(InodeStat*, utime_t, MetaSession*, UserPerm const&)+0x8d3) [0x5623150b4de3] 8: (Client::insert_readdir_results(MetaRequest*, MetaSession*, Inode*, Inode*)+0x7e1) [0x5623150b5ee1] 9: (Client::insert_trace(MetaRequest*, MetaSession*)+0x102f) [0x5623150b844f] 10: (Client::handle_client_reply(boost::intrusive_ptr const&)+0x2e3) [0x5623150b8ef3] 11: (Client::ms_dispatch2(boost::intrusive_ptr const&)+0x50d) [0x5623150d457d] 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x468) [0x7f17153962a8] 13: (DispatchQueue::entry()+0x657) [0x7f1715393847] 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1715458d51] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f1715c6f609] 16: clone() 2023-05-10T22:34:39.789 INFO:scrape:1 jobs: ['7268514'] 2023-05-10T22:34:39.789 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-10T22:34:39.789 INFO:scrape: 2023-05-10T22:34:39.790 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-05-10T22:34:39.790 INFO:scrape:2 jobs: ['7268264', '7268343'] 2023-05-10T22:34:39.790 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-05-10T22:34:39.791 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'distro/{ubuntu_20.04}', 'fs/thrash/multifs/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'rhel_8}', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}'] 2023-05-10T22:34:39.791 INFO:scrape: 2023-05-10T22:34:39.791 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-05-10T22:34:39.791 INFO:scrape:1 jobs: ['7268295'] 2023-05-10T22:34:39.792 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-05-10T22:34:39.792 INFO:scrape: 2023-05-10T22:34:39.792 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/fs/fscrypt.sh none dbench 2023-05-10T22:34:39.793 INFO:scrape:1 jobs: ['7268440'] 2023-05-10T22:34:39.793 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}'] 2023-05-10T22:34:39.793 INFO:scrape: 2023-05-10T22:34:39.793 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2023-05-10T22:34:39.794 INFO:scrape:1 jobs: ['7268313'] 2023-05-10T22:34:39.794 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-05-10T22:34:39.794 INFO:scrape: 2023-05-10T22:34:39.806 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/ failed: 30 dead: 11 running: 1 waiting: 0 queued: 0 passed: 158 Fail ================================================================= [7268254] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:51:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268254/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268254/ "1683685660.2707245 mon.a (mon.0) 464 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7268264] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 06:37:40 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268264/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268264/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f156ef0f60c44dc1a3b0a0732bd7bfdc Command failed (workunit test suites/fsstress.sh) on smithi008 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7268265] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:17:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268265/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41c91ab4734b43ada7a7da84cffa1379 Command failed on smithi093 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7268280] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:01:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268280/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c643494817f4063ae449f7cbeafb2c2 saw valgrind issues [7268295] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:19:14 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268295/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5cb1050b23f4608bfd7f0fd47c7fc8c Command failed (workunit test fs/test_python.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7268302] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:19:49 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268302/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ecf1958e74364b18bfcdc152c046bb18 Command failed (workunit test suites/dbench.sh) on smithi071 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7268313] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268313/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66a3543162f1412d9ad7d5add0108470 Command failed (workunit test fs/quota/quota.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7268318] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:18:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268318/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268318/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95b768bf3a554942863fe108e64faccf Command failed on smithi177 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7268343] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 06:44:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268343/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed5ec8a8aa904481a23d1c9d2af37841 Command failed (workunit test suites/fsstress.sh) on smithi063 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7268356] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 01:28:31 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268356/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d50c262d88684122924773a62ec1fbd0 Command failed (workunit test kernel_untar_build.sh) on smithi169 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7268376] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:33:52 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268376/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268376/ reached maximum tries (50) after waiting for 300 seconds [7268392] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:42:46 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268392/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5196f8c1a324206b657df1b75da0d53 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi145 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7268413] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:00:10 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268413/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e191a2e20574dcf9d9b898c5041b0e1 saw valgrind issues [7268415] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:37:15 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268415/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8b2461139bb470389f3d1b3fe68eda4 Command failed (workunit test kernel_untar_build.sh) on smithi073 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7268416] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:49:39 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268416/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268416/ Found coredumps on ubuntu@smithi118.front.sepia.ceph.com [7268428] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:19:21 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268428/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268428/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fffa4a2de3bc47eb83d4285799b300c0 Command failed (workunit test suites/dbench.sh) on smithi158 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7268433] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:18:04 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268433/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268433/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93fafb04365a4d03b01939f716e2bf33 Command failed on smithi187 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7268440] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 06:20:22 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268440/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268440/ sentry: https://sentry.ceph.com/organizations/ceph/?query=12fc65638dc642a2a6a57034c029a481 Command failed (workunit test fs/fscrypt.sh) on smithi186 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' [7268441] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268441/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268441/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f3d34d812664154bc3ea8d3844c36fc Command failed on smithi098 with status 8: 'rm -f /tmp/linux-image.deb && echo linux-image-6.3.0-gd8cc0e0d9358_6.3.0-gd8cc0e0d9358-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testi ng/d8cc0e0d93583718825c83a107c1ec8808dfb2dc/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-gd8cc0e0d9358/ --input-file=-' [7268454] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268454/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268454/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b78d6cb9f6164ba0819bb65c22197c2d Command failed on smithi103 with status 8: 'rm -f /tmp/linux-image.deb && echo linux-image-6.3.0-gd8cc0e0d9358_6.3.0-gd8cc0e0d9358-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testi ng/d8cc0e0d93583718825c83a107c1ec8808dfb2dc/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-gd8cc0e0d9358/ --input-file=-' [7268463] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:20:30 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268463/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee3cdaae5a714a86ba3eaadc657b5e6d Command failed (workunit test suites/dbench.sh) on smithi169 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7268465] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:04 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268465/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8cae13aba2474131a1ed75483ea834c3 Command failed on smithi120 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7268468] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 02:03:11 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268468/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268468/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c267dcb88404344bcfd02c3eeb17661 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7268472] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:14:52 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268472/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268472/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7e6250048ac44e7b9cc9d66cbf634ec Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7268492] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 01:59:24 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268492/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268492/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b472c1a7637540dbbb1ef13fc934f22b saw valgrind issues [7268495] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:46:47 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268495/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268495/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e69d979cc7924f7d8493f4802fd809f4 Command failed (workunit test libcephfs/test.sh) on smithi092 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7268499] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:41:14 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268499/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268499/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da16e3c28c28436686567c6c7fa9ef68 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7268506] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268506/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43f8f0cf1eff4ce2ab866e294b3b4847 Command failed on smithi090 with status 8: 'rm -f /tmp/linux-image.deb && echo linux-image-6.3.0-gd8cc0e0d9358_6.3.0-gd8cc0e0d9358-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testi ng/d8cc0e0d93583718825c83a107c1ec8808dfb2dc/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-gd8cc0e0d9358/ --input-file=-' [7268510] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268510/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1e8214d43e9496c8a5d2ab7b280d931 Command failed on smithi100 with status 8: 'rm -f /tmp/linux-image.deb && echo linux-image-6.3.0-gd8cc0e0d9358_6.3.0-gd8cc0e0d9358-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testi ng/d8cc0e0d93583718825c83a107c1ec8808dfb2dc/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-gd8cc0e0d9358/ --input-file=-' [7268514] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:22:22 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268514/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268514/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c49fee4121f42819496e2606980ebc0 Command failed (workunit test suites/ffsb.sh) on smithi121 with status 77: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e28f4bf0df5d53a504fe40e32a78dd63506e2421 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh' Dead ================================================================= [7268258] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268258/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268258/ hit max job timeout [7268298] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268298/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268298/ hit max job timeout [7268307] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268307/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268307/ hit max job timeout [7268323] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268323/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268323/ hit max job timeout [7268369] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268369/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268369/ hit max job timeout [7268402] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268402/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268402/ hit max job timeout [7268405] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268405/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268405/ hit max job timeout [7268418] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268418/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268418/ hit max job timeout [7268432] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268432/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268432/ hit max job timeout [7268453] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268453/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268453/ hit max job timeout [7268462] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268462/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268462/ hit max job timeout Running ================================================================= [7268520] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268520/ Pass ================================================================= [7268255] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:30:01 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268255/ [7268257] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:32 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268257/ [7268260] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:30:54 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268260/ [7268262] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:15:48 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268262/ [7268267] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:32:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268267/ [7268269] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:53:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268269/ [7268270] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:28:49 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268270/ [7268272] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:22:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268272/ [7268274] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:43:34 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268274/ [7268275] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:29:47 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268275/ [7268277] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:44:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268277/ [7268279] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:28:26 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268279/ [7268282] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:25:06 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268282/ [7268283] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:43:44 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268283/ [7268285] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:06:20 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268285/ [7268287] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:22:06 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268287/ [7268288] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:34:42 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268288/ [7268290] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:17:45 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268290/ [7268292] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:27:36 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268292/ [7268293] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:04 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268293/ [7268297] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:45 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268297/ [7268300] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:17:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268300/ [7268303] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:52:39 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268303/ [7268305] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:36:56 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268305/ [7268308] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:29:57 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268308/ [7268310] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:40:09 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268310/ [7268311] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:24:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268311/ [7268315] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:32:20 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268315/ [7268316] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:15:18 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268316/ [7268320] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:57:40 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268320/ [7268321] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:16:53 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268321/ [7268325] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:32:01 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268325/ [7268326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:10:31 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268326/ [7268328] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:56 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268328/ [7268329] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:43 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268329/ [7268331] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:17:43 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268331/ [7268333] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268333/ [7268334] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:31 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268334/ [7268336] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:18:27 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268336/ [7268338] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:34:01 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268338/ [7268339] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:21:43 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268339/ [7268341] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:23:10 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268341/ [7268344] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:03 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268344/ [7268346] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:18:30 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268346/ [7268348] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:22:20 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268348/ [7268349] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268349/ [7268351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:15:09 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268351/ [7268353] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:33:53 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268353/ [7268354] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:23:13 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268354/ [7268357] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:45:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268357/ [7268359] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:14:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268359/ [7268361] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 02:15:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268361/ [7268362] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:25:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268362/ [7268364] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:41 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268364/ [7268366] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:21:15 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268366/ [7268367] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:52:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268367/ [7268371] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:27:18 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268371/ [7268372] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:19:01 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268372/ [7268374] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} time: 02:07:33 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268374/ [7268377] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:58:50 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268377/ [7268379] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:55:37 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268379/ [7268380] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:32 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268380/ [7268382] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:50:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268382/ [7268384] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:26:30 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268384/ [7268385] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:12:33 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268385/ [7268387] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 02:17:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268387/ [7268389] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:49:53 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268389/ [7268390] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:40:27 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268390/ [7268394] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:14 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268394/ [7268395] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:10:38 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268395/ [7268397] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:09:55 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268397/ [7268398] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:27:38 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268398/ [7268400] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:39:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268400/ [7268403] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:14:34 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268403/ [7268407] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:29:45 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268407/ [7268408] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:47 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268408/ [7268410] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 07:03:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268410/ [7268412] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:33:58 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268412/ [7268420] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:22 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268420/ [7268421] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:28:06 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268421/ [7268423] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268423/ [7268424] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:17:58 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268424/ [7268426] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:58:30 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268426/ [7268429] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:19 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268429/ [7268430] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:25 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268430/ [7268431] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:22:46 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268431/ [7268434] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:26:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268434/ [7268435] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:06:40 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268435/ [7268436] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:18:26 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268436/ [7268437] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:24:11 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268437/ [7268438] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:15:48 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268438/ [7268439] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:19:03 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268439/ [7268442] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:57 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268442/ [7268443] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:43:44 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268443/ [7268444] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:33:17 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268444/ [7268445] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:29:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268445/ [7268446] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:25:28 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268446/ [7268447] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:21:22 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268447/ [7268448] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:18:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268448/ [7268449] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:29 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268449/ [7268450] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268450/ [7268451] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:26:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268451/ [7268452] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268452/ [7268455] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:25:47 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268455/ [7268456] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:27 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268456/ [7268457] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:41 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268457/ [7268458] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:19:54 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268458/ [7268459] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:18:02 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268459/ [7268460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268460/ [7268461] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:40:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268461/ [7268464] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:27:22 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268464/ [7268466] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:40 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268466/ [7268467] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:54:39 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268467/ [7268469] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:24 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268469/ [7268470] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:17:34 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268470/ [7268471] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:54 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268471/ [7268473] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:39:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268473/ [7268474] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:36:02 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268474/ [7268475] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268475/ [7268476] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:30:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268476/ [7268477] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:14:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268477/ [7268478] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:21:42 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268478/ [7268479] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 02:06:36 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268479/ [7268480] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:31:03 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268480/ [7268481] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:29:06 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268481/ [7268482] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:44 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268482/ [7268483] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:41:19 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268483/ [7268484] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:24:27 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268484/ [7268485] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:16:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268485/ [7268486] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:18:18 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268486/ [7268487] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 00:51:06 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268487/ [7268488] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:47:46 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268488/ [7268489] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:23:42 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268489/ [7268490] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:19:55 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268490/ [7268491] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:37:24 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268491/ [7268493] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:55 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268493/ [7268494] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:20:13 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268494/ [7268496] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:04 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268496/ [7268497] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:32 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268497/ [7268498] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:37 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268498/ [7268500] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:37:15 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268500/ [7268501] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268501/ [7268502] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:36:47 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268502/ [7268503] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:52:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268503/ [7268504] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:53:36 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268504/ [7268505] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:39:25 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268505/ [7268507] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:40:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268507/ [7268508] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:38:18 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268508/ [7268509] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:10:36 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268509/ [7268511] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:25:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268511/ [7268512] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:38 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268512/ [7268513] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:25:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268513/ [7268515] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:41:17 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268515/ [7268516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:20:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268516/ [7268517] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:19:28 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268517/ [7268518] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:51:34 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268518/ [7268519] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:01:38 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268519/ [7268521] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:05:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_12:06:46-fs-wip-vshankar-testing-20230509.090020-testing-default-smithi/7268521/