2023-08-09T05:53:22.568 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-09T05:53:22.569 INFO:teuthology.results:subset: '273/1024' 2023-08-09T05:53:22.569 INFO:teuthology.results:seed: '3759' 2023-08-09T06:36:47.515 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-09T06:36:47.776 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-09T15:38:30.181 INFO:teuthology.results:Tests finished! gathering results... 2023-08-09T15:38:33.627 INFO:scrape:Found 33 jobs 2023-08-09T15:38:34.021 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364214/teuthology.log 2023-08-09T15:38:39.783 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:39.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:39.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:39.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:39.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:39.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:39.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:39.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:39.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:39.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:39.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:39.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:39.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:39.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:39.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:39.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:39.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:39.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:39.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.803 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:39.804 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:39.804 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:39.805 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.805 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:39.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.808 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.808 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:39.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:39.810 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:39.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:39.812 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.812 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:39.812 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:39.813 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:39.814 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.814 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.814 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:39.815 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.816 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:39.816 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:39.817 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.817 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.818 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.819 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:39.820 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.820 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:39.821 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.821 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.822 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:39.822 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.823 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.823 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:39.824 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:39.825 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.825 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.825 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.826 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.826 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:39.827 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.827 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.828 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.828 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.829 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.829 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.830 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.830 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:39.830 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.831 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.831 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.832 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.832 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.833 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.833 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:39.833 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:39.834 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:39.834 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.835 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:39.835 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.836 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:39.836 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:39.837 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.837 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.837 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.838 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.839 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.839 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.840 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.840 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:39.841 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:39.841 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.842 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.842 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:39.843 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:39.843 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.844 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:39.845 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:39.845 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:39.846 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:39.846 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:39.847 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:39.848 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:39.848 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:39.849 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:39.850 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.853 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:39.854 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:39.854 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:39.855 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:39.855 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:39.856 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:39.857 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.857 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.858 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.858 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:39.859 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:39.859 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:39.860 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:39.861 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:39.861 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:39.863 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.864 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:39.865 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:39.866 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.868 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:39.869 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:39.869 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.870 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.871 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.871 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.872 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:39.873 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:39.873 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:39.874 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:39.874 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:39.875 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:39.919 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:39.921 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:39.923 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:39.924 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:39.925 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:39.926 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.926 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.927 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.928 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:39.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:39.930 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:39.930 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.931 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.931 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.932 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:39.932 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.933 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:39.933 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:39.933 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.934 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:39.934 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.935 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.935 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:39.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:39.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:39.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:39.938 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:39.939 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:39.939 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.939 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.940 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:39.940 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:39.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:39.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.942 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.942 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:39.943 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.943 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.943 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.944 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.944 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.945 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:39.945 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.946 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:39.946 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:39.946 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.947 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:39.947 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.948 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:39.948 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:39.949 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:39.949 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.949 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.950 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:39.950 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.951 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:39.951 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:39.952 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.952 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.952 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.953 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:39.953 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.954 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:39.954 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.955 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.955 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:39.955 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.956 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.956 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:39.957 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:39.957 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.958 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.958 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.958 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:39.959 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:39.959 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.960 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.960 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.961 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.961 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:39.961 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:39.962 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.962 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:39.963 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.963 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.963 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.964 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.964 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.965 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.965 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:39.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:39.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:39.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.967 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:39.967 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.968 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:39.968 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:39.969 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:39.969 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:39.969 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.970 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.970 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:39.971 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:39.971 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.972 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:39.972 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:39.972 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:39.973 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:39.973 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:39.974 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:39.975 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:39.975 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:39.975 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:39.976 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:39.976 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:39.977 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:39.977 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:39.978 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:39.978 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:39.978 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:39.980 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.982 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:39.983 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:39.984 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:39.984 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:39.985 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:39.985 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:39.986 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.986 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.987 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.987 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:39.988 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:39.988 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:39.989 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:39.989 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:39.990 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:39.991 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.993 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:39.993 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:39.995 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.996 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:39.997 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:39.997 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.998 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.998 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:39.999 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:39.999 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.000 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.000 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.001 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.001 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.002 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.035 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.037 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.037 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.038 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.038 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.039 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.039 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.040 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.040 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.041 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.041 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.042 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.042 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.042 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.043 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.043 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.044 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.044 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.045 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.045 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.046 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.046 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.046 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.047 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.047 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.048 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.048 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.049 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.049 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.049 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.050 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.050 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.051 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.051 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.052 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.052 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.052 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.053 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.053 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.054 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.054 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.055 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.055 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.055 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.056 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.056 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.057 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.057 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.058 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.058 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.058 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.059 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.059 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.060 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.060 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.061 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.061 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.062 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.062 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.063 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.063 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.063 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.064 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.064 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.065 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.065 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.066 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.066 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.066 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.067 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.067 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.068 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.068 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.069 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.069 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.069 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.070 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.070 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.071 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.071 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.072 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.072 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.072 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.073 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.073 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.074 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.074 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.075 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.075 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.076 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.076 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.077 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.077 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.077 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.078 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.078 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.079 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.079 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.080 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.080 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.081 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.082 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.082 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.083 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.083 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.084 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.085 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.085 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.086 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.086 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.088 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.088 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.089 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.089 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.092 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.097 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.098 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.099 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.099 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.100 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.101 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.102 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.103 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.105 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.106 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.107 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.108 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.109 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.110 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.111 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.112 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.112 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.113 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.113 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.114 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.149 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.150 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.151 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.152 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.152 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.153 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.153 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.154 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.154 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.155 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.155 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.156 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.156 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.157 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.158 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.158 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.159 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.159 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.160 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.160 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.161 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.161 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.162 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.162 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.163 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.163 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.164 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.164 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.165 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.165 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.166 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.166 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.166 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.167 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.167 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.168 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.168 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.169 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.169 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.169 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.170 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.170 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.171 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.171 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.172 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.172 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.173 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.173 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.173 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.174 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.175 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.175 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.175 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.176 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.176 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.177 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.177 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.178 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.178 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.179 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.179 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.180 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.180 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.181 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.181 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.182 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.182 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.183 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.183 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.184 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.184 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.185 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.185 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.186 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.186 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.187 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.187 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.188 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.188 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.189 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.189 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.189 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.190 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.190 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.191 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.192 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.192 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.192 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.193 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.193 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.194 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.194 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.194 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.195 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.195 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.196 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.196 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.197 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.197 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.197 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.198 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.198 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.199 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.199 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.199 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.200 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.200 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.201 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.201 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.202 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.202 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.203 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.203 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.204 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.204 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.205 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.205 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.206 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.206 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.207 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.207 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.208 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.208 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.210 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.212 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.213 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.214 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.214 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.215 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.215 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.216 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.217 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.217 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.218 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.218 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.219 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.219 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.220 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.220 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.222 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.223 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.224 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.225 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.227 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.227 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.228 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.229 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.229 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.230 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.230 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.231 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.231 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.232 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.233 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.233 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.265 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.266 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.267 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.267 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.268 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.269 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.269 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.269 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.270 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.271 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.272 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.273 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.273 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.274 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.274 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.275 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.275 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.275 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.276 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.276 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.277 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.277 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.279 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.280 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.280 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.280 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.281 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.281 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.282 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.282 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.282 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.283 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.283 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.284 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.284 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.285 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.285 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.285 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.286 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.286 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.287 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.287 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.288 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.288 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.288 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.289 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.289 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.290 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.290 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.290 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.291 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.292 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.292 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.292 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.293 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.293 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.294 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.294 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.295 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.295 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.295 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.296 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.296 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.297 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.297 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.297 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.298 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.298 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.299 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.299 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.300 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.300 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.300 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.301 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.301 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.302 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.302 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.302 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.303 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.303 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.304 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.304 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.305 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.305 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.305 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.306 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.306 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.307 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.307 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.307 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.308 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.308 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.309 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.309 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.310 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.310 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.310 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.311 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.311 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.312 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.312 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.312 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.313 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.313 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.314 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.314 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.316 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.316 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.316 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.317 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.317 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.318 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.318 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.319 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.319 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.320 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.320 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.320 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.321 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.322 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.325 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.325 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.326 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.326 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.327 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.327 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.328 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.328 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.329 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.329 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.330 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.330 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.331 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.331 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.332 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.333 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.334 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.335 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.336 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.338 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.338 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.339 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.339 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.340 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.340 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.341 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.341 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.342 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.342 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.343 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.343 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.380 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.381 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.382 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.383 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.383 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.384 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.384 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.385 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.385 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.386 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.386 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.386 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.387 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.387 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.388 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.388 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.389 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.389 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.390 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.390 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.391 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.391 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.392 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.392 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.392 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.393 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.393 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.394 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.394 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.395 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.395 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.395 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.396 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.396 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.397 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.397 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.397 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.398 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.398 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.399 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.399 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.400 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.400 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.400 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.401 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.401 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.402 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.402 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.403 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.403 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.403 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.404 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.405 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.405 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.406 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.406 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.407 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.407 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.408 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.408 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.409 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.409 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.410 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.410 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.413 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.413 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.414 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.415 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.415 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.416 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.416 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.417 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.418 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.418 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.419 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.419 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.420 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.421 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.421 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.422 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.422 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.423 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.423 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.423 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.424 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.424 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.425 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.425 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.426 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.426 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.426 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.427 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.427 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.428 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.428 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.429 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.429 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.429 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.430 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.430 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.431 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.431 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.432 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.432 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.433 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.434 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.435 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.435 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.435 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.436 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.436 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.437 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.437 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.439 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.441 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.442 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.443 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.443 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.444 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.444 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.445 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.445 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.446 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.447 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.447 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.448 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.449 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.449 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.450 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.451 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.452 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.453 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.455 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.456 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.457 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.457 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.458 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.458 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.459 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.459 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.460 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.460 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.461 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.461 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.462 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.498 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.500 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.501 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.502 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.503 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.504 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.505 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.506 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.507 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.507 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.508 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.508 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.509 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.509 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.510 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.510 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.511 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.511 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.512 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.513 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.513 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.514 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.515 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.515 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.515 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.516 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.516 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.517 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.517 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.518 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.518 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.519 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.519 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.520 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.520 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.521 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.521 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.522 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.523 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.523 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.524 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.524 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.525 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.525 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.526 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.526 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.526 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.527 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.528 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.528 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.528 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.529 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.529 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.530 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.530 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.531 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.531 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.532 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.532 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.533 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.533 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.533 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.534 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.534 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.535 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.535 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.536 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.536 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.536 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.537 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.538 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.538 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.539 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.539 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.540 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.540 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.541 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.541 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.542 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.542 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.543 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.543 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.544 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.544 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.544 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.545 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.546 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.546 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.547 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.547 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.548 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.548 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.549 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.550 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.550 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.551 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.551 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.552 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.552 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.553 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.553 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.554 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.554 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.555 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.555 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.555 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.556 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.556 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.557 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.557 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.559 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.560 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.560 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.562 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.563 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.565 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.568 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.569 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.569 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.570 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.571 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.571 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.572 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.573 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.573 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.574 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.575 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.576 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.576 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.578 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.579 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.580 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.581 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.583 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.583 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.584 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.585 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.585 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.586 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.586 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.587 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.587 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.588 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.588 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.622 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.623 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.624 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.625 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.626 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.626 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.627 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.627 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.628 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.628 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.629 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.629 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.630 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.631 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.631 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.632 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.632 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.633 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.633 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.634 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.634 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.635 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.635 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.635 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.636 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.636 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.637 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.637 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.638 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.638 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.639 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.639 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.640 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.640 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.640 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.641 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.641 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.642 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.642 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.643 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.643 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.644 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.644 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.645 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.645 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.645 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.646 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.646 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.647 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.647 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.648 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.648 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.649 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.649 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.649 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.650 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.650 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.651 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.652 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.652 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.653 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.653 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.654 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.654 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.655 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.655 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.656 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.656 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.657 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.657 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.658 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.658 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.659 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.659 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.660 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.660 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.661 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.661 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.662 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.662 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.663 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.664 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.664 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.665 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.665 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.666 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.666 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.667 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.667 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.668 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.668 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.668 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.669 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.670 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.670 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.671 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.671 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.672 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.672 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.673 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.673 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.674 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.674 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.675 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.676 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.676 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.677 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.677 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.678 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.678 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.679 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.679 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.680 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.680 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.681 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.682 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.682 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.683 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.683 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.684 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.684 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.685 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.685 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.686 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.687 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.690 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.690 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.691 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.692 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.693 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.693 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.694 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.695 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.695 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.696 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.696 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.697 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.698 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.699 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.699 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.701 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.703 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.703 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.705 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.706 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.707 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.708 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.708 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.709 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.709 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.710 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.710 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.711 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.711 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.712 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.712 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.747 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.748 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.749 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.749 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.751 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.752 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.753 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.754 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.755 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.756 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.756 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.757 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.758 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.759 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.760 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.760 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.761 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.761 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.762 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.763 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.764 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.764 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.765 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.768 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.769 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.769 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.770 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.771 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.771 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.772 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.772 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.773 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.774 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.778 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.778 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.781 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.785 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.788 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.792 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.795 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.798 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.802 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.804 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.805 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.807 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.808 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.808 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.810 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.810 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.812 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.813 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.814 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.815 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.816 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.818 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.818 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.819 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.819 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.820 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.821 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.821 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.821 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.822 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.822 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.823 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.823 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:40.856 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-5396-ge318c197 (e318c19728255ff705e0d6553aab0c6a87b22d64) reef (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x128) [0x7fb3a3b234af] , 2: /usr/lib/ceph/libceph-common.so.2(+0x32366b) [0x7fb3a3b2366b] 2023-08-09T15:38:40.857 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.328+0000 7f3bae94b700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b94004e20 0x7f3b940052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bad949700 1 --2- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 0x7f3b94004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.328+0000 7f3bae14a700 1 -- 172.21.15.2:0/2470276836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3ba814a340 msgr2=0x7f3b94004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-09T15:38:40.858 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:29.744+0000 7ff8a6ddc700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00a87c0 con 0x7ff8a00b28f0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:29.744+0000 7ff8a55d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8a00b8e90 0x7ff8a00aa0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45256/0 (socket says 172.21.15.2:45256) 2023-08-09T15:38:40.858 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:30.391 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 8 , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 -- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 msgr2=0x7f99ec067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:30.388+0000 7f99fffff700 1 --2- 172.21.15.2:0/1384396096 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7f99ec0656f0 0x7f99ec067bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a001415d0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-08-09T15:38:40.859 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800e070 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800b910 con 0x7fab440b1540 , 2023-08-09T06:50:31.040+0000 7fab437fe700 1 -- 172.21.15.2:0/3919971431 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3800bcc0 con 0x7fab440b1540 2023-08-09T15:38:40.860 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.448+0000 7f7bff7fe700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7c080afd20 0x7f7c080acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 -- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 msgr2=0x7f7c080ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:31.448+0000 7f7c0cb14700 1 --2- 172.21.15.2:0/335633916 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7c080a9d90 0x7f7c080ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.861 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.861 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.861 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", , 2023-08-09T06:50:31.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", , 2023-08-09T06:50:31.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.862 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:31.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.862 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:31.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-08-09T15:38:40.863 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:31.848 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { , 2023-08-09T06:50:31.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-08-09T15:38:40.863 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:31.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:31.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.864 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", , 2023-08-09T06:50:31.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:31.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.864 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", , 2023-08-09T06:50:31.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.865 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", , 2023-08-09T06:50:31.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.865 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", , 2023-08-09T06:50:31.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-08-09T15:38:40.865 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.011 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", , 2023-08-09T06:50:32.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.866 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.866 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.063 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { , 2023-08-09T06:50:32.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-08-09T15:38:40.867 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.867 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.105 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-08-09T15:38:40.868 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.868 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", , 2023-08-09T06:50:32.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.868 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.869 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.869 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.870 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.870 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2023-08-09T06:50:32.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", , 2023-08-09T06:50:32.250 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-08-09T15:38:40.871 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.271 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-08-09T15:38:40.871 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-08-09T15:38:40.872 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.872 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.333 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", , 2023-08-09T06:50:32.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.873 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.873 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { , 2023-08-09T06:50:32.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { , 2023-08-09T06:50:32.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-08-09T15:38:40.873 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:32.400 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.874 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.432 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-08-09T15:38:40.874 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.875 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.875 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.876 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.876 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.877 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:32.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.877 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.877 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.878 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", , 2023-08-09T06:50:32.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-08-09T15:38:40.878 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", , 2023-08-09T06:50:32.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.879 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.879 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { , 2023-08-09T06:50:32.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-08-09T15:38:40.880 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.880 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.719 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.881 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.881 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.764 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.765 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.882 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ , 2023-08-09T06:50:32.787 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-08-09T15:38:40.882 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:32.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-08-09T15:38:40.883 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:32.828 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.883 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:32.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.883 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:32.875 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-08-09T15:38:40.884 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:32.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.884 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:32.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.885 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:32.940 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-08-09T15:38:40.885 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:32.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.886 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:32.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:32.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.886 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", , 2023-08-09T06:50:33.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.887 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-08-09T15:38:40.887 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.888 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.076 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-08-09T15:38:40.888 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", , 2023-08-09T06:50:33.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.889 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", , 2023-08-09T06:50:33.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.889 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-08-09T15:38:40.889 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.890 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.890 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { , 2023-08-09T06:50:33.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-08-09T15:38:40.891 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.891 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", , 2023-08-09T06:50:33.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.892 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.892 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.893 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { , 2023-08-09T06:50:33.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", , 2023-08-09T06:50:33.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-08-09T15:38:40.893 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.893 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.894 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.894 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.895 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", , 2023-08-09T06:50:33.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.895 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", , 2023-08-09T06:50:33.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-08-09T15:38:40.896 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-08-09T15:38:40.896 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.897 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-08-09T15:38:40.897 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.898 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.898 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.899 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.899 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.900 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.900 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-08-09T15:38:40.901 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", , 2023-08-09T06:50:33.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-08-09T15:38:40.901 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] , 2023-08-09T06:50:33.704 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:33.705 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-08-09T15:38:40.902 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.725 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.902 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "error", , 2023-08-09T06:50:33.745 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-08-09T15:38:40.903 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: ], , 2023-08-09T06:50:33.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.903 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, , 2023-08-09T06:50:33.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-08-09T15:38:40.904 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-08-09T15:38:40.904 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-08-09T15:38:40.905 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-08-09T15:38:40.905 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:33.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.906 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:33.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.906 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:33.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-08-09T15:38:40.907 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", , 2023-08-09T06:50:33.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:33.931 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-08-09T15:38:40.907 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:33.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.908 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" , 2023-08-09T06:50:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-08-09T15:38:40.908 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", , 2023-08-09T06:50:33.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", , 2023-08-09T06:50:33.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-08-09T15:38:40.909 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", , 2023-08-09T06:50:34.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-08-09T15:38:40.909 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", , 2023-08-09T06:50:34.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-08-09T15:38:40.910 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", , 2023-08-09T06:50:34.052 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-08-09T15:38:40.910 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2023-08-09T06:50:34.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-08-09T15:38:40.911 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], , 2023-08-09T06:50:34.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-08-09T15:38:40.911 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ , 2023-08-09T06:50:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", , 2023-08-09T06:50:34.113 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-08-09T15:38:40.911 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.318 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 325, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 4, , 2023-08-09T06:50:34.319 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 2.408427040, 2023-08-09T15:38:40.912 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_backtrace_fetch": 508, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "openino_peer_discover": 0, , 2023-08-09T06:50:34.347 INFO:teuthology.orchestra.run.smithi002.stdout: "root_rfiles": 44, 2023-08-09T15:38:40.913 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "req_getattr_latency": { , 2023-08-09T06:50:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 1392, , 2023-08-09T06:50:34.372 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 8.309027109, 2023-08-09T15:38:40.913 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 28, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025563294, , 2023-08-09T06:50:34.393 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000912974 2023-08-09T15:38:40.913 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, , 2023-08-09T06:50:34.414 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-08-09T15:38:40.914 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_ops_high_water": 96, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing": 0, , 2023-08-09T06:50:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: "pq_executing_high_water": 32, 2023-08-09T15:38:40.915 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { , 2023-08-09T06:50:34.637 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 80, , 2023-08-09T06:50:34.638 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 78, 2023-08-09T15:38:40.915 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.672 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, , 2023-08-09T06:50:34.673 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2023-08-09T15:38:40.917 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 msgr2=0x7facb006fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 --2- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7facb0144160 0x7facb006fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:50:34.888+0000 7facb5cde700 1 -- 172.21.15.2:0/3138196529 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facb00af180 msgr2=0x7facb0147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.919 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 msgr2=0x7f53a01deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 --2- 172.21.15.2:0/2532422973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a013cd20 0x7f53a01deb30 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f5390000c00 tx=0x7f5390004770 comp rx=0 tx=0).stop , 2023-08-09T06:50:35.564+0000 7f53a742c700 1 -- 172.21.15.2:0/2532422973 shutdown_connections 2023-08-09T15:38:40.920 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:36.276+0000 7fcbfaffd700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc0406f300 0x7fcc04147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc0413e730 0x7fcc040b2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) , 2023-08-09T06:50:36.276+0000 7fcc03fff700 1 -- 172.21.15.2:0/3158668541 learned_addr learned my addr 172.21.15.2:0/3158668541 (peer_addr_for_me v1:172.21.15.2:0/0) 2023-08-09T15:38:40.921 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fddd8071cf0 0x7fddd81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fddd80ade70 0x7fddd81daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-08-09T06:50:37.752+0000 7fdddca19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fddd80b1830 0x7fddd81db4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.921 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:37.996+0000 7f6eb0d49700 1 --2- 172.21.15.2:0/1770242582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6eac071e40 0x7f6eac0ad220 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ba0 tx=0x7f6e9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0045c0 con 0x7f6eac071e40 , 2023-08-09T06:50:37.996+0000 7f6ea1ffb700 1 -- 172.21.15.2:0/1770242582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c01e070 con 0x7f6eac071e40 2023-08-09T15:38:40.922 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:38.412+0000 7fd188a5e700 1 -- 172.21.15.2:0/1858961049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1840b38a0 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 -- 172.21.15.2:0/1858961049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 84962+0+0 (secure 0 0 0) 0x7fd17c005470 con 0x7fd184149d50 , 2023-08-09T06:50:38.416+0000 7fd172ffd700 1 --2- 172.21.15.2:0/1858961049 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd164065720 0x7fd164067be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.922 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2023-08-09T06:50:38.893 INFO:tasks.ceph:Scrubbing osd.2 , 2023-08-09T06:50:38.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-08-09T15:38:40.923 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972800b910 con 0x7f97380b4110 , 2023-08-09T06:50:39.312+0000 7f973659c700 1 -- 172.21.15.2:0/2459629771 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800bcc0 con 0x7f97380b4110 , 2023-08-09T06:50:39.316+0000 7f973c861700 1 -- 172.21.15.2:0/2459629771 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f97380b4110 msgr2=0x7f9738149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.924 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:39.732+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 learned_addr learned my addr 172.21.15.2:0/1793139873 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:50:39.732+0000 7f904e7fc700 1 --2- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f90501445d0 0x7f90501deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:50:39.736+0000 7f904effd700 1 -- 172.21.15.2:0/1793139873 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9050147c90 msgr2=0x7f9050142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.924 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7ff5f40656f0 0x7ff5f4067bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff604000f60 tx=0x7ff604009040 comp rx=0 tx=0).stop , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 -- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 msgr2=0x7ff6101ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:40.180+0000 7ff615bb7700 1 --2- 172.21.15.2:0/1782154886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff61014a360 0x7ff6101ded40 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cbd0 tx=0x7ff60800cee0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.925 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900af830 con 0x7f9d90145940 , 2023-08-09T06:50:40.644+0000 7f9d9648e700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d900b2590 con 0x7f9d9014b870 , 2023-08-09T06:50:40.644+0000 7f9d94c8b700 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d90145940 0x7f9d90145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.925 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 Processor -- start , 2023-08-09T06:50:41.056+0000 7f4c44c51700 1 -- start start , 2023-08-09T06:50:41.060+0000 7f4c44c51700 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4c40070e40 0x7f4c401cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-09T15:38:40.926 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0005430 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f5701984700 1 -- 172.21.15.2:0/2095473025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc1df880 con 0x7f56fc0a9d90 , 2023-08-09T06:50:41.300+0000 7f56f97fa700 1 -- 172.21.15.2:0/2095473025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00055a0 con 0x7f56fc0a9d90 2023-08-09T15:38:40.926 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 -- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 msgr2=0x7f3cdc148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:41.744+0000 7f3ce3487700 1 --2- 172.21.15.2:0/2672108048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0b52e0 0x7f3cdc148800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800b370 tx=0x7f3cd8004af0 comp rx=0 tx=0).stop , 2023-08-09T06:50:41.744+0000 7f3ce1a24700 1 -- 172.21.15.2:0/2672108048 reap_dead start 2023-08-09T15:38:40.927 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 >> v1:172.21.15.2:6789/0 conn(0x7f96a40aea60 legacy=0x7f96a4146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 -- 172.21.15.2:0/2441283432 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4145d90 con 0x7f96a40b0d60 , 2023-08-09T06:50:42.372+0000 7f96a8df5700 1 --2- 172.21.15.2:0/2441283432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96a40b0d60 0x7f96a413c470 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f969c009d90 tx=0x7f969c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9e0da49ae62a server_cookie=0 in_seq=0 out_seq=0 2023-08-09T15:38:40.928 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146750 con 0x7fad100b4110 , 2023-08-09T06:50:42.628+0000 7fad1618b700 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10146540 con 0x7fad1014a190 , 2023-08-09T06:50:42.628+0000 7fad14988700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad10140390 0x7fad101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-09T15:38:40.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.236+0000 7fd4ca7fc700 1 -- 172.21.15.2:0/81649446 <== mgr.4101 v2:172.21.15.2:6832/4153827865 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308811 (secure 0 0 0) 0x7fd4d000bd30 con 0x7fd4b80656f0 , 2023-08-09T06:50:43.315 INFO:teuthology.orchestra.run.smithi002.stdout: , 2023-08-09T06:50:43.240+0000 7fd4dac5a700 1 -- 172.21.15.2:0/81649446 >> [v2:172.21.15.2:6832/4153827865,v1:172.21.15.2:6833/4153827865] conn(0x7fd4b80656f0 msgr2=0x7fd4b8067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.931 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-08-09T06:46:03.361686+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=3, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.369 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:50:43.370 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.931 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.811 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) , 2023-08-09T06:51:03.812 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-08-09T06:46:04.368227+0000 time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=46, tm_sec=4, tm_wday=2, tm_yday=221, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=8, tm_mday=9, tm_hour=6, tm_min=50, tm_sec=36, tm_wday=2, tm_yday=221, tm_isdst=0) 2023-08-09T15:38:40.933 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 learned_addr learned my addr 172.21.15.2:0/177636733 (peer_addr_for_me v2:172.21.15.2:0/0) , 2023-08-09T06:51:24.035+0000 7f7d5caca700 1 --2- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 0x7f7d581de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:24.035+0000 7f7d577fe700 1 -- 172.21.15.2:0/177636733 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7d5813f400 msgr2=0x7f7d581de220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.934 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d001f070 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2e4ff9700 1 -- 172.21.15.2:0/2182246067 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00036c0 con 0x7fc2e8140390 , 2023-08-09T06:51:44.479+0000 7fc2ecd0e700 1 -- 172.21.15.2:0/2182246067 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8142ee0 con 0x7fc2e8140390 2023-08-09T15:38:40.935 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools , 2023-08-09T06:51:45.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json , 2023-08-09T06:51:45.339+0000 7f7b73c9e700 1 Processor -- start 2023-08-09T15:38:40.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fa9280722f0 tx=0x7fa92000b330 comp rx=0 tx=0).stop , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 -- 172.21.15.2:0/2687007240 shutdown_connections , 2023-08-09T06:51:45.783+0000 7fa92f400700 1 --2- 172.21.15.2:0/2687007240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa928140560 0x7fa928150a20 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.936 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.223+0000 7f9e0a59c700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e0c0701b0 0x7f9e0c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 -- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 msgr2=0x7f9e0c1cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:46.223+0000 7f9e0b59e700 1 --2- 172.21.15.2:0/3762608096 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9e0c06fdd0 0x7f9e0c1cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f421c06bfd0 tx=0x7f4214004af0 comp rx=0 tx=0).stop , 2023-08-09T06:51:46.875+0000 7f4221825700 1 -- 172.21.15.2:0/2952973954 shutdown_connections , 2023-08-09T06:51:46.875+0000 7f4221825700 1 --2- 172.21.15.2:0/2952973954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f421c1474e0 0x7f421c1db580 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-09T15:38:40.937 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5af400b9c0 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5af959a700 1 -- 172.21.15.2:0/1883012582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5afc071810 , 2023-08-09T06:51:48.095+0000 7f5afb59e700 1 -- 172.21.15.2:0/1883012582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5afc071810 msgr2=0x7f5afc146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-09T15:38:40.938 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 msgr2=0x7f8b640caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 --2- 172.21.15.2:0/1594541054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b64039ec0 0x7f8b640caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-08-09T06:51:49.127+0000 7f8b7278a700 1 -- 172.21.15.2:0/1594541054 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c04eef0 con 0x7f8b640c6e00 2023-08-09T15:38:40.938 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:33.094 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2023-08-09T06:52:33.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com , 2023-08-09T06:52:33.574 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-09T15:38:40.939 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log , /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log , /var/log/ceph/ceph-client.admin.61303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-08-09T15:38:40.939 WARNING:scrape:Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log , /var/log/ceph/ceph-client.admin.61989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log , 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2023-08-09T15:38:40.940 WARNING:scrape:Ignoring malparsed backtrace: , /var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2023-08-09T15:38:40.940 WARNING:scrape:Ignoring malparsed backtrace: 2023-08-09T06:52:59.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-08-09T06:53:00.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2023-08-09T06:53:00.034 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-09T15:38:47.228 WARNING:scrape:Misunderstood line: 2023-08-09T15:38:50.626 INFO:scrape:Found 15 distinct failure reasons 2023-08-09T15:38:50.627 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi169 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-08-09T15:38:50.628 INFO:scrape:2 jobs: ['7364206', '7364188'] 2023-08-09T15:38:50.628 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-08-09T15:38:50.629 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-08-09T15:38:50.629 INFO:scrape: 2023-08-09T15:38:50.629 INFO:scrape:Dead: None 2023-08-09T15:38:50.629 INFO:scrape:1 jobs: ['7364214'] 2023-08-09T15:38:50.630 INFO:scrape: 2023-08-09T15:38:50.630 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2023-08-09T15:38:50.631 INFO:scrape:1 jobs: ['7364207'] 2023-08-09T15:38:50.631 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-08-09T15:38:50.631 INFO:scrape: 2023-08-09T15:38:50.632 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-08-09T15:38:50.632 INFO:scrape:1 jobs: ['7364196'] 2023-08-09T15:38:50.632 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-08-09T15:38:50.633 INFO:scrape: 2023-08-09T15:38:50.633 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2023-08-09T15:38:50.633 INFO:scrape:2 jobs: ['7364197', '7364218'] 2023-08-09T15:38:50.634 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-08-09T15:38:50.634 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-08-09T15:38:50.634 INFO:scrape: 2023-08-09T15:38:50.634 INFO:scrape:Failure: Command failed on smithi069 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-08-09T15:38:50.635 INFO:scrape:4 jobs: ['7364201', '7364195', '7364213', '7364208'] 2023-08-09T15:38:50.635 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-08-09T15:38:50.635 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-08-09T15:38:50.636 INFO:scrape: 2023-08-09T15:38:50.636 INFO:scrape:Failure: Command failed (workunit test suites/iozone.sh) on smithi115 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' 2023-08-09T15:38:50.636 INFO:scrape:1 jobs: ['7364203'] 2023-08-09T15:38:50.637 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-08-09T15:38:50.637 INFO:scrape: 2023-08-09T15:38:50.637 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-08-09T15:38:50.637 INFO:scrape:1 jobs: ['7364217'] 2023-08-09T15:38:50.638 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-08-09T15:38:50.638 INFO:scrape: 2023-08-09T15:38:50.638 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi002 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-08-09T15:38:50.639 INFO:scrape:1 jobs: ['7364204'] 2023-08-09T15:38:50.639 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-08-09T15:38:50.639 INFO:scrape: 2023-08-09T15:38:50.639 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds 2023-08-09T15:38:50.640 INFO:scrape:1 jobs: ['7364215'] 2023-08-09T15:38:50.640 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}'] 2023-08-09T15:38:50.640 INFO:scrape: 2023-08-09T15:38:50.641 INFO:scrape:Failure: "2023-08-09T06:45:22.522390+0000 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 140556.922133 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-08-09T15:38:50.641 INFO:scrape:1 jobs: ['7364205'] 2023-08-09T15:38:50.641 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-08-09T15:38:50.642 INFO:scrape: 2023-08-09T15:38:50.642 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-08-09T15:38:50.642 INFO:scrape:1 jobs: ['7364210'] 2023-08-09T15:38:50.643 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-08-09T15:38:50.643 INFO:scrape: 2023-08-09T15:38:50.643 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-08-09T15:38:50.644 INFO:scrape:1 jobs: ['7364202'] 2023-08-09T15:38:50.644 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-08-09T15:38:50.644 INFO:scrape: 2023-08-09T15:38:50.644 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-08-09T15:38:50.645 INFO:scrape:1 jobs: ['7364209'] 2023-08-09T15:38:50.645 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-08-09T15:38:50.645 INFO:scrape: 2023-08-09T15:38:50.646 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-08-09T15:38:50.646 INFO:scrape:1 jobs: ['7364200'] 2023-08-09T15:38:50.646 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-08-09T15:38:50.647 INFO:scrape: 2023-08-09T15:38:50.649 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/ failed: 19 dead: 1 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [7364188] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:27:03 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364188/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2820a49f72f4009af9245d22e14b433 Command failed (workunit test kernel_untar_build.sh) on smithi125 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7364195] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:39 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364195/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0cd291b4d7944d88a00ca2c0ab4b167b Command failed on smithi162 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7364196] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 09:19:03 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364196/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45a3ef6084fd458aa138492ce9f0da23 Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7364197] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:17:40 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364197/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=234320704c184f37b286247b4f87d7ae Command failed (workunit test fs/quota/quota.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7364200] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:49:21 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364200/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbd59d064503406397c40afff8507624 Command failed (workunit test suites/dbench.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7364201] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:57 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364201/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364201/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f17ad501cab479ea55760975ad59840 Command failed on smithi069 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7364202] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:30:46 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364202/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2cb83d6ae40d47e887fa8c923d65cdba saw valgrind issues [7364203] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:25:36 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364203/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38df9d89614946738b782fb410e8d9d0 Command failed (workunit test suites/iozone.sh) on smithi115 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' [7364204] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:19:53 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364204/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f97a7aa79c244a63b4c85f5b905ca4a5 Command failed (workunit test libcephfs/test.sh) on smithi002 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7364205] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:25:10 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364205/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364205/ "2023-08-09T06:45:22.522390+0000 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 140556.922133 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7364206] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:24:25 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364206/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364206/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d0a92af70304dbd863cd2191ca40d05 Command failed (workunit test kernel_untar_build.sh) on smithi169 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7364207] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:16:46 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364207/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ea04b0169794a9f84d633037e4a8920 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7364208] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:03 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364208/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1feb799c43a6423c857325820247df08 Command failed on smithi093 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7364209] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:16:20 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364209/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbc92c61f20143ca818c5861f4be1d6f Command failed (workunit test fs/test_python.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7364210] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:17:08 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364210/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364210/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7c1a23327374d2bbb645b1745fe3c94 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7364213] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:10 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364213/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7f31c9f999f48ac912f86428aaae107 Command failed on smithi125 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7364215] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:15:46 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364215/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364215/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7364217] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:40:27 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364217/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff76472d0c1949a78aeb763f2ff51061 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7364218] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:18:24 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364218/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364218/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e6578319e474d7f83eddaa36c0b1358 Command failed (workunit test fs/quota/quota.sh) on smithi090 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e318c19728255ff705e0d6553aab0c6a87b22d64 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh' Dead ================================================================= [7364214] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364214/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364214/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7364186] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:21:11 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364186/ [7364187] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:17:32 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364187/ [7364189] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:24:33 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364189/ [7364190] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:35:04 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364190/ [7364191] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:20 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364191/ [7364192] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:09:24 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364192/ [7364193] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:25:31 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364193/ [7364194] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:18:52 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364194/ [7364198] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:38 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364198/ [7364199] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:22:00 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364199/ [7364211] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 01:00:47 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364211/ [7364212] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:13:20 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364212/ [7364216] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:27:12 info: http://pulpito.ceph.com/vshankar-2023-08-09_05:51:38-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7364216/