Found 3 jobs Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f29eed3c14a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f29eed3c364] Ignoring malparsed backtrace: 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-01-15T06:53:43.965 INFO:tasks.workunit:Stopping ['fs/test_python.sh'] on client.0... , 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run.smithi199:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 Ignoring malparsed backtrace: 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44b00629b0 0x7f44b00a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 -- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 msgr2=0x7f44b00a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 0x7f44b00a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:45.170+0000 7faccd92e700 1 -- 172.21.15.199:0/3007788719 shutdown_connections , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7faca80581d0 0x7faca805a690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc80ac1e0 0x7facc80a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:46.637+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c009920 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c009ca0 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb684643700 1 -- 172.21.15.199:0/663141147 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fb67c0ab610 msgr2=0x7fb67c0a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:46.807+0000 7fd61b7fe700 1 -- 172.21.15.199:0/3292017515 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60800ec10 con 0x7fd6000044d0 , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 -- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 msgr2=0x7fd600004950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 --2- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 0x7fd600004950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd60800bb60 tx=0x7fd608008040).stop Ignoring malparsed backtrace: 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 msgr2=0x7f40b014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 --2- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 0x7f40b014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40b00a9240 msgr2=0x7f40b0137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.478+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 --> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.479+0000 7f68ae7fc700 1 -- 172.21.15.199:0/1818373123 <== mgr.4101 v2:172.21.15.190:6800/37012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.480+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f68a00581d0 msgr2=0x7f68a005a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 msgr2=0x7f2ef8061b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 --2- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 0x7f2ef8061b20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f2ef8058200 msgr2=0x7f2ef805a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 --2- 172.21.15.199:0/1361116367 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f49c4149c70 0x7f49c40a4e90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 >> 172.21.15.199:0/1361116367 conn(0x7f49c4060360 msgr2=0x7f49c413bbc0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:47.953+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.199:6789/0 conn(0x7f83640b1f30 legacy=0x7f83640ab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.190:6789/0 conn(0x7f83640a8e90 legacy=0x7f836413b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83640adcb0 con 0x7f836413bf40 Ignoring malparsed backtrace: 2021-01-15T06:53:48.574+0000 7f8a67fff700 1 -- 172.21.15.199:0/1269882982 learned_addr learned my addr 172.21.15.199:0/1269882982 (peer_addr_for_me v2:172.21.15.199:0/0) , 2021-01-15T06:53:48.574+0000 7f8a66ffd700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a680acb40 0x7f8a680a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.574+0000 7f8a677fe700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a68062ca0 0x7f8a6813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f78d40a9af0 tx=0x7f78c8008040).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2021-01-15T06:53:48.735+0000 7f78c27fc700 1 -- 172.21.15.199:0/1879962590 <== mon.1 v2:172.21.15.199:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f78c4047720 con 0x7f78d4062990 Ignoring malparsed backtrace: 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 Processor -- start , 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 -- start start , 2021-01-15T06:53:49.205+0000 7f09cb81b700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7f09c413bf40 0x7f09c4146d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.520+0000 7f5901110700 1 -- start start , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc067a90 0x7f58fc13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f58fc067e70 0x7f58fc14a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe040a2e00 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbdeaffd700 1 -- 172.21.15.199:0/3565618221 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc007170 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe040a1100 con 0x7fbe040ada30 Ignoring malparsed backtrace: 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 >> 172.21.15.199:0/1992364206 conn(0x7f2020060360 msgr2=0x7f2020060890 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:50.441+0000 7fd33b7fe700 1 -- 172.21.15.199:0/3435411477 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd334011610 con 0x7fd33c0aafb0 , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 -- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 msgr2=0x7fd33c0a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 --2- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 0x7fd33c0a5c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd334009d50 tx=0x7fd334009290).stop Ignoring malparsed backtrace: 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 >> v1:172.21.15.190:6789/0 conn(0x7f0ffc0a8e90 legacy=0x7f0ffc13b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0ada50 con 0x7f0ffc0b1f30 , 2021-01-15T06:54:10.777+0000 7f1002021700 1 --2- 172.21.15.199:0/1444929167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ffc13bf40 0x7f0ffc145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! Ignoring malparsed backtrace: 2021-01-15T06:54:31.105+0000 7f8a0a91f700 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040a6be0 con 0x7f8a040629b0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.199:3300/0 says I am v2:172.21.15.199:57390/0 (socket says 172.21.15.199:57390) Ignoring malparsed backtrace: 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.417+0000 7f8d5fd2d700 1 -- 172.21.15.199:0/2448182109 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58045190 con 0x7f8d5814a9c0 , 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8d54006e20 tx=0x7f8d54016000).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:54:32.115+0000 7f07e53bf700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07e00b1f30 0x7f07e00ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f07e013bd20 0x7f07e013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.199:35230/0 (socket says 172.21.15.199:35230) , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 -- 172.21.15.199:0/1835036666 learned_addr learned my addr 172.21.15.199:0/1835036666 (peer_addr_for_me v1:172.21.15.199:0/0) Ignoring malparsed backtrace: 2021-01-15T06:54:32.499+0000 7feae6871700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7feae00b2240 0x7feae014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e350 con 0x7feae00b1e60 , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e570 con 0x7feae00b2240 Ignoring malparsed backtrace: 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 -- 172.21.15.199:0/2262236411 <== mon.1 v2:172.21.15.199:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5cd4071760 con 0x7f5cdc138350 , 2021-01-15T06:54:32.814+0000 7f5ce19a4700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:54:34.776+0000 7efdb349c700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7efdac13b010 0x7efdac13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v1:172.21.15.199:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac0a9980 con 0x7efdac0ab610 , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdac080150 con 0x7efdac0a9240 Ignoring malparsed backtrace: 2021-01-15T06:54:35.746+0000 7f97effff700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40044d0 0x7f97d4004950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f54ab700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 0x7f97d4004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f4caa700 1 -- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 msgr2=0x7f97d4004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:54:37.144 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-15T06:54:37.183 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-15T06:54:37.183 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== , 2021-01-15T06:54:41.834 INFO:teuthology.orchestra.run.smithi190.stdout:^- hv01.front.sepia.ceph.com 3 6 377 38 -4176us[-4176us] +/- 107ms Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f29eed3c14a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f29eed3c364] Ignoring malparsed backtrace: 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-01-15T06:53:43.965 INFO:tasks.workunit:Stopping ['fs/test_python.sh'] on client.0... , 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run.smithi199:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 Ignoring malparsed backtrace: 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44b00629b0 0x7f44b00a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 -- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 msgr2=0x7f44b00a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 0x7f44b00a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:45.170+0000 7faccd92e700 1 -- 172.21.15.199:0/3007788719 shutdown_connections , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7faca80581d0 0x7faca805a690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc80ac1e0 0x7facc80a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:46.637+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c009920 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c009ca0 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb684643700 1 -- 172.21.15.199:0/663141147 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fb67c0ab610 msgr2=0x7fb67c0a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:46.807+0000 7fd61b7fe700 1 -- 172.21.15.199:0/3292017515 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60800ec10 con 0x7fd6000044d0 , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 -- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 msgr2=0x7fd600004950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 --2- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 0x7fd600004950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd60800bb60 tx=0x7fd608008040).stop Ignoring malparsed backtrace: 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 msgr2=0x7f40b014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 --2- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 0x7f40b014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40b00a9240 msgr2=0x7f40b0137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.478+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 --> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.479+0000 7f68ae7fc700 1 -- 172.21.15.199:0/1818373123 <== mgr.4101 v2:172.21.15.190:6800/37012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.480+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f68a00581d0 msgr2=0x7f68a005a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 msgr2=0x7f2ef8061b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 --2- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 0x7f2ef8061b20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f2ef8058200 msgr2=0x7f2ef805a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 --2- 172.21.15.199:0/1361116367 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f49c4149c70 0x7f49c40a4e90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 >> 172.21.15.199:0/1361116367 conn(0x7f49c4060360 msgr2=0x7f49c413bbc0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:47.953+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.199:6789/0 conn(0x7f83640b1f30 legacy=0x7f83640ab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.190:6789/0 conn(0x7f83640a8e90 legacy=0x7f836413b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83640adcb0 con 0x7f836413bf40 Ignoring malparsed backtrace: 2021-01-15T06:53:48.574+0000 7f8a67fff700 1 -- 172.21.15.199:0/1269882982 learned_addr learned my addr 172.21.15.199:0/1269882982 (peer_addr_for_me v2:172.21.15.199:0/0) , 2021-01-15T06:53:48.574+0000 7f8a66ffd700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a680acb40 0x7f8a680a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.574+0000 7f8a677fe700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a68062ca0 0x7f8a6813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f78d40a9af0 tx=0x7f78c8008040).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2021-01-15T06:53:48.735+0000 7f78c27fc700 1 -- 172.21.15.199:0/1879962590 <== mon.1 v2:172.21.15.199:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f78c4047720 con 0x7f78d4062990 Ignoring malparsed backtrace: 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 Processor -- start , 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 -- start start , 2021-01-15T06:53:49.205+0000 7f09cb81b700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7f09c413bf40 0x7f09c4146d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.520+0000 7f5901110700 1 -- start start , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc067a90 0x7f58fc13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f58fc067e70 0x7f58fc14a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe040a2e00 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbdeaffd700 1 -- 172.21.15.199:0/3565618221 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc007170 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe040a1100 con 0x7fbe040ada30 Ignoring malparsed backtrace: 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 >> 172.21.15.199:0/1992364206 conn(0x7f2020060360 msgr2=0x7f2020060890 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:50.441+0000 7fd33b7fe700 1 -- 172.21.15.199:0/3435411477 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd334011610 con 0x7fd33c0aafb0 , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 -- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 msgr2=0x7fd33c0a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 --2- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 0x7fd33c0a5c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd334009d50 tx=0x7fd334009290).stop Ignoring malparsed backtrace: 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 >> v1:172.21.15.190:6789/0 conn(0x7f0ffc0a8e90 legacy=0x7f0ffc13b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0ada50 con 0x7f0ffc0b1f30 , 2021-01-15T06:54:10.777+0000 7f1002021700 1 --2- 172.21.15.199:0/1444929167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ffc13bf40 0x7f0ffc145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! Ignoring malparsed backtrace: 2021-01-15T06:54:31.105+0000 7f8a0a91f700 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040a6be0 con 0x7f8a040629b0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.199:3300/0 says I am v2:172.21.15.199:57390/0 (socket says 172.21.15.199:57390) Ignoring malparsed backtrace: 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.417+0000 7f8d5fd2d700 1 -- 172.21.15.199:0/2448182109 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58045190 con 0x7f8d5814a9c0 , 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8d54006e20 tx=0x7f8d54016000).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:54:32.115+0000 7f07e53bf700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07e00b1f30 0x7f07e00ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f07e013bd20 0x7f07e013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.199:35230/0 (socket says 172.21.15.199:35230) , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 -- 172.21.15.199:0/1835036666 learned_addr learned my addr 172.21.15.199:0/1835036666 (peer_addr_for_me v1:172.21.15.199:0/0) Ignoring malparsed backtrace: 2021-01-15T06:54:32.499+0000 7feae6871700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7feae00b2240 0x7feae014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e350 con 0x7feae00b1e60 , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e570 con 0x7feae00b2240 Ignoring malparsed backtrace: 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 -- 172.21.15.199:0/2262236411 <== mon.1 v2:172.21.15.199:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5cd4071760 con 0x7f5cdc138350 , 2021-01-15T06:54:32.814+0000 7f5ce19a4700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:54:34.776+0000 7efdb349c700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7efdac13b010 0x7efdac13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v1:172.21.15.199:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac0a9980 con 0x7efdac0ab610 , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdac080150 con 0x7efdac0a9240 Ignoring malparsed backtrace: 2021-01-15T06:54:35.746+0000 7f97effff700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40044d0 0x7f97d4004950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f54ab700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 0x7f97d4004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f4caa700 1 -- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 msgr2=0x7f97d4004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:54:37.144 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-15T06:54:37.183 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-15T06:54:37.183 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== , 2021-01-15T06:54:41.834 INFO:teuthology.orchestra.run.smithi190.stdout:^- hv01.front.sepia.ceph.com 3 6 377 38 -4176us[-4176us] +/- 107ms Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f29eed3c14a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f29eed3c364] Ignoring malparsed backtrace: 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-01-15T06:53:43.965 INFO:tasks.workunit:Stopping ['fs/test_python.sh'] on client.0... , 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run.smithi199:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 Ignoring malparsed backtrace: 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44b00629b0 0x7f44b00a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 -- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 msgr2=0x7f44b00a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 0x7f44b00a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:45.170+0000 7faccd92e700 1 -- 172.21.15.199:0/3007788719 shutdown_connections , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7faca80581d0 0x7faca805a690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc80ac1e0 0x7facc80a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:46.637+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c009920 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c009ca0 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb684643700 1 -- 172.21.15.199:0/663141147 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fb67c0ab610 msgr2=0x7fb67c0a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:46.807+0000 7fd61b7fe700 1 -- 172.21.15.199:0/3292017515 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60800ec10 con 0x7fd6000044d0 , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 -- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 msgr2=0x7fd600004950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 --2- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 0x7fd600004950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd60800bb60 tx=0x7fd608008040).stop Ignoring malparsed backtrace: 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 msgr2=0x7f40b014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 --2- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 0x7f40b014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40b00a9240 msgr2=0x7f40b0137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.478+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 --> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.479+0000 7f68ae7fc700 1 -- 172.21.15.199:0/1818373123 <== mgr.4101 v2:172.21.15.190:6800/37012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.480+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f68a00581d0 msgr2=0x7f68a005a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 msgr2=0x7f2ef8061b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 --2- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 0x7f2ef8061b20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f2ef8058200 msgr2=0x7f2ef805a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 --2- 172.21.15.199:0/1361116367 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f49c4149c70 0x7f49c40a4e90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 >> 172.21.15.199:0/1361116367 conn(0x7f49c4060360 msgr2=0x7f49c413bbc0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:47.953+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.199:6789/0 conn(0x7f83640b1f30 legacy=0x7f83640ab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.190:6789/0 conn(0x7f83640a8e90 legacy=0x7f836413b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83640adcb0 con 0x7f836413bf40 Ignoring malparsed backtrace: 2021-01-15T06:53:48.574+0000 7f8a67fff700 1 -- 172.21.15.199:0/1269882982 learned_addr learned my addr 172.21.15.199:0/1269882982 (peer_addr_for_me v2:172.21.15.199:0/0) , 2021-01-15T06:53:48.574+0000 7f8a66ffd700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a680acb40 0x7f8a680a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.574+0000 7f8a677fe700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a68062ca0 0x7f8a6813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f78d40a9af0 tx=0x7f78c8008040).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2021-01-15T06:53:48.735+0000 7f78c27fc700 1 -- 172.21.15.199:0/1879962590 <== mon.1 v2:172.21.15.199:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f78c4047720 con 0x7f78d4062990 Ignoring malparsed backtrace: 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 Processor -- start , 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 -- start start , 2021-01-15T06:53:49.205+0000 7f09cb81b700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7f09c413bf40 0x7f09c4146d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.520+0000 7f5901110700 1 -- start start , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc067a90 0x7f58fc13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f58fc067e70 0x7f58fc14a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe040a2e00 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbdeaffd700 1 -- 172.21.15.199:0/3565618221 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc007170 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe040a1100 con 0x7fbe040ada30 Ignoring malparsed backtrace: 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 >> 172.21.15.199:0/1992364206 conn(0x7f2020060360 msgr2=0x7f2020060890 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:50.441+0000 7fd33b7fe700 1 -- 172.21.15.199:0/3435411477 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd334011610 con 0x7fd33c0aafb0 , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 -- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 msgr2=0x7fd33c0a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 --2- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 0x7fd33c0a5c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd334009d50 tx=0x7fd334009290).stop Ignoring malparsed backtrace: 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 >> v1:172.21.15.190:6789/0 conn(0x7f0ffc0a8e90 legacy=0x7f0ffc13b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0ada50 con 0x7f0ffc0b1f30 , 2021-01-15T06:54:10.777+0000 7f1002021700 1 --2- 172.21.15.199:0/1444929167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ffc13bf40 0x7f0ffc145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! Ignoring malparsed backtrace: 2021-01-15T06:54:31.105+0000 7f8a0a91f700 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040a6be0 con 0x7f8a040629b0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.199:3300/0 says I am v2:172.21.15.199:57390/0 (socket says 172.21.15.199:57390) Ignoring malparsed backtrace: 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.417+0000 7f8d5fd2d700 1 -- 172.21.15.199:0/2448182109 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58045190 con 0x7f8d5814a9c0 , 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8d54006e20 tx=0x7f8d54016000).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:54:32.115+0000 7f07e53bf700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07e00b1f30 0x7f07e00ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f07e013bd20 0x7f07e013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.199:35230/0 (socket says 172.21.15.199:35230) , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 -- 172.21.15.199:0/1835036666 learned_addr learned my addr 172.21.15.199:0/1835036666 (peer_addr_for_me v1:172.21.15.199:0/0) Ignoring malparsed backtrace: 2021-01-15T06:54:32.499+0000 7feae6871700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7feae00b2240 0x7feae014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e350 con 0x7feae00b1e60 , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e570 con 0x7feae00b2240 Ignoring malparsed backtrace: 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 -- 172.21.15.199:0/2262236411 <== mon.1 v2:172.21.15.199:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5cd4071760 con 0x7f5cdc138350 , 2021-01-15T06:54:32.814+0000 7f5ce19a4700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:54:34.776+0000 7efdb349c700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7efdac13b010 0x7efdac13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v1:172.21.15.199:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac0a9980 con 0x7efdac0ab610 , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdac080150 con 0x7efdac0a9240 Ignoring malparsed backtrace: 2021-01-15T06:54:35.746+0000 7f97effff700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40044d0 0x7f97d4004950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f54ab700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 0x7f97d4004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f4caa700 1 -- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 msgr2=0x7f97d4004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:54:37.144 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-15T06:54:37.183 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-15T06:54:37.183 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== , 2021-01-15T06:54:41.834 INFO:teuthology.orchestra.run.smithi190.stdout:^- hv01.front.sepia.ceph.com 3 6 377 38 -4176us[-4176us] +/- 107ms Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f29eed3c14a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f29eed3c364] Ignoring malparsed backtrace: 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 134 , 2021-01-15T06:53:43.965 INFO:tasks.workunit:Stopping ['fs/test_python.sh'] on client.0... , 2021-01-15T06:53:43.965 DEBUG:teuthology.orchestra.run.smithi199:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 Ignoring malparsed backtrace: 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f44b00629b0 0x7f44b00a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 -- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 msgr2=0x7f44b00a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:44.688+0000 7f44b50ff700 1 --2- 172.21.15.199:0/759167193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b00ac1e0 0x7f44b00a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:45.170+0000 7faccd92e700 1 -- 172.21.15.199:0/3007788719 shutdown_connections , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7faca80581d0 0x7faca805a690 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:45.170+0000 7faccd92e700 1 --2- 172.21.15.199:0/3007788719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7facc80ac1e0 0x7facc80a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:53:46.637+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c009920 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb6813dd700 1 -- 172.21.15.199:0/663141147 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c009ca0 con 0x7fb67c0ab610 , 2021-01-15T06:53:46.638+0000 7fb684643700 1 -- 172.21.15.199:0/663141147 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fb67c0ab610 msgr2=0x7fb67c0a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:46.807+0000 7fd61b7fe700 1 -- 172.21.15.199:0/3292017515 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60800ec10 con 0x7fd6000044d0 , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 -- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 msgr2=0x7fd600004950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:46.808+0000 7fd622fce700 1 --2- 172.21.15.199:0/3292017515 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd6000044d0 0x7fd600004950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd60800bb60 tx=0x7fd608008040).stop Ignoring malparsed backtrace: 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 msgr2=0x7f40b014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 --2- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40b00ab610 0x7f40b014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.122+0000 7f40b4e42700 1 -- 172.21.15.199:0/2512157591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40b00a9240 msgr2=0x7f40b0137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.478+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 --> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.479+0000 7f68ae7fc700 1 -- 172.21.15.199:0/1818373123 <== mgr.4101 v2:172.21.15.190:6800/37012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68c00a8200 con 0x7f68a00581d0 , 2021-01-15T06:53:47.480+0000 7f68c87b6700 1 -- 172.21.15.199:0/1818373123 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f68a00581d0 msgr2=0x7f68a005a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 msgr2=0x7f2ef8061b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 --2- 172.21.15.199:0/2994414379 >> [v2:172.21.15.199:6808/37364,v1:172.21.15.199:6809/37364] conn(0x7f2ef805f6c0 0x7f2ef8061b20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.638+0000 7f2f1f0cc700 1 -- 172.21.15.199:0/2994414379 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f2ef8058200 msgr2=0x7f2ef805a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 --2- 172.21.15.199:0/1361116367 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f49c4149c70 0x7f49c40a4e90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:53:47.952+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 >> 172.21.15.199:0/1361116367 conn(0x7f49c4060360 msgr2=0x7f49c413bbc0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:47.953+0000 7f49cad0e700 1 -- 172.21.15.199:0/1361116367 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.199:6789/0 conn(0x7f83640b1f30 legacy=0x7f83640ab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 >> v1:172.21.15.190:6789/0 conn(0x7f83640a8e90 legacy=0x7f836413b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:48.256+0000 7f83637fe700 1 -- 172.21.15.199:0/1621650191 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83640adcb0 con 0x7f836413bf40 Ignoring malparsed backtrace: 2021-01-15T06:53:48.574+0000 7f8a67fff700 1 -- 172.21.15.199:0/1269882982 learned_addr learned my addr 172.21.15.199:0/1269882982 (peer_addr_for_me v2:172.21.15.199:0/0) , 2021-01-15T06:53:48.574+0000 7f8a66ffd700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a680acb40 0x7f8a680a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.574+0000 7f8a677fe700 1 --2- 172.21.15.199:0/1269882982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a68062ca0 0x7f8a6813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:53:48.731+0000 7f78d1d9b700 1 --2- 172.21.15.199:0/1879962590 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f78b00581d0 0x7f78b005a690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f78d40a9af0 tx=0x7f78c8008040).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2021-01-15T06:53:48.735+0000 7f78c27fc700 1 -- 172.21.15.199:0/1879962590 <== mon.1 v2:172.21.15.199:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f78c4047720 con 0x7f78d4062990 Ignoring malparsed backtrace: 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 Processor -- start , 2021-01-15T06:53:49.204+0000 7f09cb81b700 1 -- start start , 2021-01-15T06:53:49.205+0000 7f09cb81b700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7f09c413bf40 0x7f09c4146d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.520+0000 7f5901110700 1 -- start start , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58fc067a90 0x7f58fc13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:53:49.521+0000 7f5901110700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f58fc067e70 0x7f58fc14a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe040a2e00 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbdeaffd700 1 -- 172.21.15.199:0/3565618221 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc007170 con 0x7fbe040ada30 , 2021-01-15T06:53:49.673+0000 7fbe03fff700 1 -- 172.21.15.199:0/3565618221 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe040a1100 con 0x7fbe040ada30 Ignoring malparsed backtrace: 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 >> 172.21.15.199:0/1992364206 conn(0x7f2020060360 msgr2=0x7f2020060890 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:53:50.003+0000 7f2027bf3700 1 -- 172.21.15.199:0/1992364206 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:53:50.441+0000 7fd33b7fe700 1 -- 172.21.15.199:0/3435411477 <== mon.1 v2:172.21.15.199:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd334011610 con 0x7fd33c0aafb0 , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 -- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 msgr2=0x7fd33c0a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:53:50.441+0000 7fd34320d700 1 --2- 172.21.15.199:0/3435411477 >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7fd33c0aafb0 0x7fd33c0a5c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd334009d50 tx=0x7fd334009290).stop Ignoring malparsed backtrace: 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 >> v1:172.21.15.190:6789/0 conn(0x7f0ffc0a8e90 legacy=0x7f0ffc13b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2021-01-15T06:54:10.777+0000 7f1001820700 1 -- 172.21.15.199:0/1444929167 --> v2:172.21.15.199:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0ada50 con 0x7f0ffc0b1f30 , 2021-01-15T06:54:10.777+0000 7f1002021700 1 --2- 172.21.15.199:0/1444929167 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ffc13bf40 0x7f0ffc145cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! Ignoring malparsed backtrace: 2021-01-15T06:54:31.105+0000 7f8a0a91f700 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040a6be0 con 0x7f8a040629b0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.105+0000 7f8a08ebc700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7f8a04138940 0x7f8a040a5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.199:3300/0 says I am v2:172.21.15.199:57390/0 (socket says 172.21.15.199:57390) Ignoring malparsed backtrace: 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:31.417+0000 7f8d5fd2d700 1 -- 172.21.15.199:0/2448182109 --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58045190 con 0x7f8d5814a9c0 , 2021-01-15T06:54:31.417+0000 7f8d5dac9700 1 --2- 172.21.15.199:0/2448182109 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f8d38058250 0x7f8d3805a710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8d54006e20 tx=0x7f8d54016000).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:54:32.115+0000 7f07e53bf700 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07e00b1f30 0x7f07e00ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f07e013bd20 0x7f07e013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.199:35230/0 (socket says 172.21.15.199:35230) , 2021-01-15T06:54:32.115+0000 7f07e5bc0700 1 -- 172.21.15.199:0/1835036666 learned_addr learned my addr 172.21.15.199:0/1835036666 (peer_addr_for_me v1:172.21.15.199:0/0) Ignoring malparsed backtrace: 2021-01-15T06:54:32.499+0000 7feae6871700 1 --2- >> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] conn(0x7feae00b2240 0x7feae014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e350 con 0x7feae00b1e60 , 2021-01-15T06:54:32.499+0000 7feae6871700 1 -- --> [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e570 con 0x7feae00b2240 Ignoring malparsed backtrace: 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:32.813+0000 7f5cc9ffb700 1 -- 172.21.15.199:0/2262236411 <== mon.1 v2:172.21.15.199:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5cd4071760 con 0x7f5cdc138350 , 2021-01-15T06:54:32.814+0000 7f5ce19a4700 1 --2- 172.21.15.199:0/2262236411 >> [v2:172.21.15.190:6800/37012,v1:172.21.15.190:6801/37012] conn(0x7f5cbc058200 0x7f5cbc05a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:54:34.776+0000 7efdb349c700 1 --2- >> v2:172.21.15.199:3300/0 conn(0x7efdac13b010 0x7efdac13b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v1:172.21.15.199:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac0a9980 con 0x7efdac0ab610 , 2021-01-15T06:54:34.776+0000 7efdb349c700 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdac080150 con 0x7efdac0a9240 Ignoring malparsed backtrace: 2021-01-15T06:54:35.746+0000 7f97effff700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97d40044d0 0x7f97d4004950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f54ab700 1 --2- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 0x7f97d4004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:54:35.746+0000 7f97f4caa700 1 -- 172.21.15.199:0/1232848663 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f00a9700 msgr2=0x7f97d4004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:54:37.144 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-15T06:54:37.183 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-15T06:54:37.183 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-15T06:54:41.833 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== , 2021-01-15T06:54:41.834 INFO:teuthology.orchestra.run.smithi190.stdout:^- hv01.front.sepia.ceph.com 3 6 377 38 -4176us[-4176us] +/- 107ms Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f36d82f814a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f36d82f8364] Ignoring malparsed backtrace: 2021-01-15T06:50:39.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, , 2021-01-15T06:50:39.606 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { , 2021-01-15T06:50:39.607 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1, Ignoring malparsed backtrace: 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 856, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 2, Ignoring malparsed backtrace: 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "finisher-MDSRank": { , 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "queue_len": 0, , 2021-01-15T06:50:39.841 INFO:teuthology.orchestra.run.smithi149.stdout: "complete_latency": { Ignoring malparsed backtrace: 2021-01-15T06:50:39.876 INFO:teuthology.orchestra.run.smithi149.stdout: "evexg": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "evexd": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "segadd": 20, Ignoring malparsed backtrace: 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "avgcount": 0, , 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "sum": 0.000000000, , 2021-01-15T06:50:39.915 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 Ignoring malparsed backtrace: 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_bytes": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_items": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_other_bytes": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "poolstat_resend": 0, , 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_active": 0, , 2021-01-15T06:50:39.998 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_send": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:40.090 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: } , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: }, Ignoring malparsed backtrace: 2021-01-15T06:50:40.408 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":22,"fsid":"992debe4-2ce0-421a-956c-91da6f0d8c35","created":"2021-01-15T06:47:45.889163+0000","modified":"2021-01-15T06:48:03.144743+0000","last_up_change":"2021-01-15T06:47:52.515218+0000","last_in_change":"2021-01-15T06:47:52.515218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-01-15T06:47:49.538942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-01-15T06:47:56.934802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2021-01-15T06:48:00.119154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2021-01-15T06:48:01.115613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fccfea96-d88f-4a39-810b-fdd874508400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":33082},{"type":"v1","addr":"172.21.15.139:6801","nonce":33082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":33082},{"type":"v1","addr":"172.21.15.139:6803","nonce":33082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":33082},{"type":"v1","addr":"172.21.15.139:6807","nonce":33082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":33082},{"type":"v1","addr":"172.21.15.139:6805","nonce":33082}]},"public_addr":"172.21.15.139:6801/33082","cluster_addr":"172.21.15.139:6803/33082","heartbeat_back_addr":"172.21.15.139:6807/33082","heartbeat_front_addr":"172.21.15.139:6805/33082","state":["exists","up"]},{"osd":1,"uuid":"33e3078f-0ed6-4140-ad99-c833950bf2cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":33083},{"type":"v1","addr":"172.21.15.139:6809","nonce":33083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":33083},{"type":"v1","addr":"172.21.15.139:6811","nonce":33083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":33083},{"type":"v1","addr":"172.21.15.139:6815","nonce":33083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":33083},{"type":"v1","addr":"172.21.15.139:6813","nonce":33083}]},"public_addr":"172.21.15.139:6809/33083","cluster_addr":"172.21.15.139:6811/33083","heartbeat_back_addr":"172.21.15.139:6815/33083","heartbeat_front_addr":"172.21.15.139:6813/33083","state":["exists","up"]},{"osd":2,"uuid":"dac8c79b-1334-4cb2-ab53-f73ac75c9f64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":33085},{"type":"v1","addr":"172.21.15.139:6817","nonce":33085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":33085},{"type":"v1","addr":"172.21.15.139:6819","nonce":33085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":33085},{"type":"v1","addr":"172.21.15.139:6823","nonce":33085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":33085},{"type":"v1","addr":"172.21.15.139:6821","nonce":33085}]},"public_addr":"172.21.15.139:6817/33085","cluster_addr":"172.21.15.139:6819/33085","heartbeat_back_addr":"172.21.15.139:6823/33085","heartbeat_front_addr":"172.21.15.139:6821/33085","state":["exists","up"]},{"osd":3,"uuid":"fc707574-d86e-4c11-afbf-c33d65bd9313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":33123},{"type":"v1","addr":"172.21.15.139:6825","nonce":33123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":33123},{"type":"v1","addr":"172.21.15.139:6827","nonce":33123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":33123},{"type":"v1","addr":"172.21.15.139:6831","nonce":33123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":33123},{"type":"v1","addr":"172.21.15.139:6829","nonce":33123}]},"public_addr":"172.21.15.139:6825/33123","cluster_addr":"172.21.15.139:6827/33123","heartbeat_back_addr":"172.21.15.139:6831/33123","heartbeat_front_addr":"172.21.15.139:6829/33123","state":["exists","up"]},{"osd":4,"uuid":"9bb8a7e3-bec0-4139-909f-91aa80808805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":33318},{"type":"v1","addr":"172.21.15.149:6817","nonce":33318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":33318},{"type":"v1","addr":"172.21.15.149:6823","nonce":33318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6829","nonce":33318},{"type":"v1","addr":"172.21.15.149:6831","nonce":33318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6825","nonce":33318},{"type":"v1","addr":"172.21.15.149:6827","nonce":33318}]},"public_addr":"172.21.15.149:6817/33318","cluster_addr":"172.21.15.149:6823/33318","heartbeat_back_addr":"172.21.15.149:6831/33318","heartbeat_front_addr":"172.21.15.149:6827/33318","state":["exists","up"]},{"osd":5,"uuid":"92842e15-59ef-4944-bea3-46ad21e8a535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":33309},{"type":"v1","addr":"172.21.15.149:6801","nonce":33309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":33309},{"type":"v1","addr":"172.21.15.149:6803","nonce":33309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":33309},{"type":"v1","addr":"172.21.15.149:6807","nonce":33309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":33309},{"type":"v1","addr":"172.21.15.149:6805","nonce":33309}]},"public_addr":"172.21.15.149:6801/33309","cluster_addr":"172.21.15.149:6803/33309","heartbeat_back_addr":"172.21.15.149:6807/33309","heartbeat_front_addr":"172.21.15.149:6805/33309","state":["exists","up"]},{"osd":6,"uuid":"5703d9ed-8fba-446c-83b1-df2f986816ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":33311},{"type":"v1","addr":"172.21.15.149:6809","nonce":33311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":33311},{"type":"v1","addr":"172.21.15.149:6811","nonce":33311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":33311},{"type":"v1","addr":"172.21.15.149:6818","nonce":33311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":33311},{"type":"v1","addr":"172.21.15.149:6813","nonce":33311}]},"public_addr":"172.21.15.149:6809/33311","cluster_addr":"172.21.15.149:6811/33311","heartbeat_back_addr":"172.21.15.149:6818/33311","heartbeat_front_addr":"172.21.15.149:6813/33311","state":["exists","up"]},{"osd":7,"uuid":"4e66a90c-1017-4524-a271-d50c1fa836c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":33320},{"type":"v1","addr":"172.21.15.149:6819","nonce":33320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6821","nonce":33320},{"type":"v1","addr":"172.21.15.149:6822","nonce":33320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":33320},{"type":"v1","addr":"172.21.15.149:6830","nonce":33320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":33320},{"type":"v1","addr":"172.21.15.149:6826","nonce":33320}]},"public_addr":"172.21.15.149:6819/33320","cluster_addr":"172.21.15.149:6822/33320","heartbeat_back_addr":"172.21.15.149:6830/33320","heartbeat_front_addr":"172.21.15.149:6826/33320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.429293+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.402855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.428641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.466464+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.430731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.447289+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.511915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.516104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-01-15T06:50:40.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-15T06:50:40.522+0000 7f8ecedcf700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 shutdown_connections , 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 wait complete. , 2021-01-15T06:50:40.805+0000 7f12c18b2700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:42.098+0000 7ff7277fe700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7280683a0 0x7ff728137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.098+0000 7ff726ffd700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7280a8e20 0x7ff728136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.099+0000 7ff727fff700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff728138250 0x7ff72813d910 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7ff72000b750 tx=0x7ff720017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d900683a0 0x7f7d90136d40 secure :-1 s=READY pgs=4437 cs=0 l=1 rev1=1 rx=0x7f7d900b16b0 tx=0x7f7d80016000).stop , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 -- 172.21.15.139:0/2284323119 shutdown_connections , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d900ae680 0x7f7d9014a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 learned_addr learned my addr 172.21.15.139:0/2781215916 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:42.943+0000 7fa04259c700 1 --2- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 0x7fa044137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 msgr2=0x7fa044137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:43.225+0000 7f337511e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3370138250 0x7f337012ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52328/0 (socket says 172.21.15.139:52328) , 2021-01-15T06:50:43.225+0000 7f337511e700 1 -- 172.21.15.139:0/3793063344 learned_addr learned my addr 172.21.15.139:0/3793063344 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:43.225+0000 7f336ffff700 1 --2- 172.21.15.139:0/3793063344 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f33700a8e20 0x7f337014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:43.400+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f1f24045f50 con 0x7f1f2c0ae1d0 , 2021-01-15T06:50:43.540+0000 7f1f313a4700 1 -- 172.21.15.139:0/2385702852 --> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f2c05d9a0 con 0x7f1f10058210 , 2021-01-15T06:50:43.541+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mgr.4101 v2:172.21.15.139:6832/32872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f2c05d9a0 con 0x7f1f10058210 Ignoring malparsed backtrace: 2021-01-15T06:50:43.825+0000 7f42dad20700 1 -- start start , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42d40683a0 0x7f42d40a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42d40ae1d0 0x7f42d40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 --2- 172.21.15.139:0/2120096777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8dc0683a0 0x7ff8dc0a8e20 unknown :-1 s=CLOSED pgs=4451 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 >> 172.21.15.139:0/2120096777 conn(0x7ff8dc080550 msgr2=0x7ff8dc067ad0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:44.110+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffafc0a67a0 0x7ffafc1319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52420/0 (socket says 172.21.15.139:52420) , 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 -- 172.21.15.139:0/175652128 learned_addr learned my addr 172.21.15.139:0/175652128 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:44.268+0000 7ffb008dc700 1 --2- 172.21.15.139:0/175652128 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffafc0a9b00 0x7ffafc148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800b950 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698025440 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6980257e0 con 0x7fa6b014ac80 Ignoring malparsed backtrace: 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f1e48058210 0x7f1e4805a6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e680ae680 0x7f1e6814e390 unknown :-1 s=CLOSED pgs=4458 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e680683a0 0x7f1e68142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 >> 172.21.15.139:0/3407257751 conn(0x7f312c080550 msgr2=0x7f312c132c70 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 shutdown_connections , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 wait complete. Ignoring malparsed backtrace: 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0009990 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009d10 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c6b2b700 1 -- 172.21.15.139:0/1779367271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62c00a9770 msgr2=0x7f62c013e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:45.588+0000 7fe98dffb700 1 -- 172.21.15.139:0/798497067 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005620 con 0x7fe97c003760 , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 -- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 msgr2=0x7fe97c003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 --2- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 0x7fe97c003b80 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7fe98000c440 tx=0x7fe99013b1e0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0007180 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77bffff700 1 -- 172.21.15.139:0/3931649689 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c00620e0 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0006520 con 0x7f77c00681a0 Ignoring malparsed backtrace: 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 Processor -- start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 -- start start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad2c004a00 0x7fad480ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.503+0000 7fc00497a700 1 -- 172.21.15.139:0/4006517727 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00005d9a0 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 -- 172.21.15.139:0/4006517727 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 5) v1 ==== 68683+0+0 (secure 0 0 0) 0x7fbff4013480 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 --2- 172.21.15.139:0/4006517727 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7fbfdc058210 0x7fbfdc05a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.998+0000 7fad9937b700 1 -- 172.21.15.139:0/1227730117 wait complete. , dumped fsmap epoch 5 , 2021-01-15T06:51:07.013 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json Ignoring malparsed backtrace: 2021-01-15T06:51:07.446+0000 7fb56d59a700 1 -- 172.21.15.139:0/3234842163 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568011610 con 0x7fb5700680b0 , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 -- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 msgr2=0x7fb5700a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 --2- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 0x7fb5700a4ce0 secure :-1 s=READY pgs=4481 cs=0 l=1 rev1=1 rx=0x7fb5700ae0f0 tx=0x7fb5680092d0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f40683a0 0x7fb0f4131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 -- 172.21.15.139:0/1561132054 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4130250 con 0x7fb0f40a8e20 , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0f40a8e20 0x7fb0f4131400 secure :-1 s=READY pgs=4484 cs=0 l=1 rev1=1 rx=0x7fb0e800ebd0 tx=0x7fb0e8006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 -- 172.21.15.139:0/1674202383 shutdown_connections , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f8cf8058240 0x7f8cf805a700 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d14137930 0x7f8d1409daf0 unknown :-1 s=CLOSED pgs=4487 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000bc80 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f84f000bde0 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011650 con 0x7f84f813eed0 Ignoring malparsed backtrace: 2021-01-15T06:51:12.618 INFO:tasks.ceph.mon.a:Stopped , 2021-01-15T06:51:12.619 DEBUG:tasks.ceph.mon.b:waiting for process to exit , 2021-01-15T06:51:12.619 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-15T06:51:33.565 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log , 2021-01-15T06:51:33.620 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-15T06:51:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f36d82f814a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f36d82f8364] Ignoring malparsed backtrace: 2021-01-15T06:50:39.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, , 2021-01-15T06:50:39.606 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { , 2021-01-15T06:50:39.607 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1, Ignoring malparsed backtrace: 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 856, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 2, Ignoring malparsed backtrace: 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "finisher-MDSRank": { , 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "queue_len": 0, , 2021-01-15T06:50:39.841 INFO:teuthology.orchestra.run.smithi149.stdout: "complete_latency": { Ignoring malparsed backtrace: 2021-01-15T06:50:39.876 INFO:teuthology.orchestra.run.smithi149.stdout: "evexg": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "evexd": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "segadd": 20, Ignoring malparsed backtrace: 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "avgcount": 0, , 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "sum": 0.000000000, , 2021-01-15T06:50:39.915 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 Ignoring malparsed backtrace: 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_bytes": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_items": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_other_bytes": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "poolstat_resend": 0, , 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_active": 0, , 2021-01-15T06:50:39.998 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_send": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:40.090 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: } , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: }, Ignoring malparsed backtrace: 2021-01-15T06:50:40.408 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":22,"fsid":"992debe4-2ce0-421a-956c-91da6f0d8c35","created":"2021-01-15T06:47:45.889163+0000","modified":"2021-01-15T06:48:03.144743+0000","last_up_change":"2021-01-15T06:47:52.515218+0000","last_in_change":"2021-01-15T06:47:52.515218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-01-15T06:47:49.538942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-01-15T06:47:56.934802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2021-01-15T06:48:00.119154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2021-01-15T06:48:01.115613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fccfea96-d88f-4a39-810b-fdd874508400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":33082},{"type":"v1","addr":"172.21.15.139:6801","nonce":33082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":33082},{"type":"v1","addr":"172.21.15.139:6803","nonce":33082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":33082},{"type":"v1","addr":"172.21.15.139:6807","nonce":33082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":33082},{"type":"v1","addr":"172.21.15.139:6805","nonce":33082}]},"public_addr":"172.21.15.139:6801/33082","cluster_addr":"172.21.15.139:6803/33082","heartbeat_back_addr":"172.21.15.139:6807/33082","heartbeat_front_addr":"172.21.15.139:6805/33082","state":["exists","up"]},{"osd":1,"uuid":"33e3078f-0ed6-4140-ad99-c833950bf2cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":33083},{"type":"v1","addr":"172.21.15.139:6809","nonce":33083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":33083},{"type":"v1","addr":"172.21.15.139:6811","nonce":33083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":33083},{"type":"v1","addr":"172.21.15.139:6815","nonce":33083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":33083},{"type":"v1","addr":"172.21.15.139:6813","nonce":33083}]},"public_addr":"172.21.15.139:6809/33083","cluster_addr":"172.21.15.139:6811/33083","heartbeat_back_addr":"172.21.15.139:6815/33083","heartbeat_front_addr":"172.21.15.139:6813/33083","state":["exists","up"]},{"osd":2,"uuid":"dac8c79b-1334-4cb2-ab53-f73ac75c9f64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":33085},{"type":"v1","addr":"172.21.15.139:6817","nonce":33085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":33085},{"type":"v1","addr":"172.21.15.139:6819","nonce":33085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":33085},{"type":"v1","addr":"172.21.15.139:6823","nonce":33085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":33085},{"type":"v1","addr":"172.21.15.139:6821","nonce":33085}]},"public_addr":"172.21.15.139:6817/33085","cluster_addr":"172.21.15.139:6819/33085","heartbeat_back_addr":"172.21.15.139:6823/33085","heartbeat_front_addr":"172.21.15.139:6821/33085","state":["exists","up"]},{"osd":3,"uuid":"fc707574-d86e-4c11-afbf-c33d65bd9313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":33123},{"type":"v1","addr":"172.21.15.139:6825","nonce":33123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":33123},{"type":"v1","addr":"172.21.15.139:6827","nonce":33123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":33123},{"type":"v1","addr":"172.21.15.139:6831","nonce":33123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":33123},{"type":"v1","addr":"172.21.15.139:6829","nonce":33123}]},"public_addr":"172.21.15.139:6825/33123","cluster_addr":"172.21.15.139:6827/33123","heartbeat_back_addr":"172.21.15.139:6831/33123","heartbeat_front_addr":"172.21.15.139:6829/33123","state":["exists","up"]},{"osd":4,"uuid":"9bb8a7e3-bec0-4139-909f-91aa80808805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":33318},{"type":"v1","addr":"172.21.15.149:6817","nonce":33318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":33318},{"type":"v1","addr":"172.21.15.149:6823","nonce":33318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6829","nonce":33318},{"type":"v1","addr":"172.21.15.149:6831","nonce":33318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6825","nonce":33318},{"type":"v1","addr":"172.21.15.149:6827","nonce":33318}]},"public_addr":"172.21.15.149:6817/33318","cluster_addr":"172.21.15.149:6823/33318","heartbeat_back_addr":"172.21.15.149:6831/33318","heartbeat_front_addr":"172.21.15.149:6827/33318","state":["exists","up"]},{"osd":5,"uuid":"92842e15-59ef-4944-bea3-46ad21e8a535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":33309},{"type":"v1","addr":"172.21.15.149:6801","nonce":33309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":33309},{"type":"v1","addr":"172.21.15.149:6803","nonce":33309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":33309},{"type":"v1","addr":"172.21.15.149:6807","nonce":33309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":33309},{"type":"v1","addr":"172.21.15.149:6805","nonce":33309}]},"public_addr":"172.21.15.149:6801/33309","cluster_addr":"172.21.15.149:6803/33309","heartbeat_back_addr":"172.21.15.149:6807/33309","heartbeat_front_addr":"172.21.15.149:6805/33309","state":["exists","up"]},{"osd":6,"uuid":"5703d9ed-8fba-446c-83b1-df2f986816ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":33311},{"type":"v1","addr":"172.21.15.149:6809","nonce":33311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":33311},{"type":"v1","addr":"172.21.15.149:6811","nonce":33311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":33311},{"type":"v1","addr":"172.21.15.149:6818","nonce":33311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":33311},{"type":"v1","addr":"172.21.15.149:6813","nonce":33311}]},"public_addr":"172.21.15.149:6809/33311","cluster_addr":"172.21.15.149:6811/33311","heartbeat_back_addr":"172.21.15.149:6818/33311","heartbeat_front_addr":"172.21.15.149:6813/33311","state":["exists","up"]},{"osd":7,"uuid":"4e66a90c-1017-4524-a271-d50c1fa836c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":33320},{"type":"v1","addr":"172.21.15.149:6819","nonce":33320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6821","nonce":33320},{"type":"v1","addr":"172.21.15.149:6822","nonce":33320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":33320},{"type":"v1","addr":"172.21.15.149:6830","nonce":33320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":33320},{"type":"v1","addr":"172.21.15.149:6826","nonce":33320}]},"public_addr":"172.21.15.149:6819/33320","cluster_addr":"172.21.15.149:6822/33320","heartbeat_back_addr":"172.21.15.149:6830/33320","heartbeat_front_addr":"172.21.15.149:6826/33320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.429293+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.402855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.428641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.466464+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.430731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.447289+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.511915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.516104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-01-15T06:50:40.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-15T06:50:40.522+0000 7f8ecedcf700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 shutdown_connections , 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 wait complete. , 2021-01-15T06:50:40.805+0000 7f12c18b2700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:42.098+0000 7ff7277fe700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7280683a0 0x7ff728137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.098+0000 7ff726ffd700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7280a8e20 0x7ff728136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.099+0000 7ff727fff700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff728138250 0x7ff72813d910 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7ff72000b750 tx=0x7ff720017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d900683a0 0x7f7d90136d40 secure :-1 s=READY pgs=4437 cs=0 l=1 rev1=1 rx=0x7f7d900b16b0 tx=0x7f7d80016000).stop , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 -- 172.21.15.139:0/2284323119 shutdown_connections , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d900ae680 0x7f7d9014a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 learned_addr learned my addr 172.21.15.139:0/2781215916 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:42.943+0000 7fa04259c700 1 --2- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 0x7fa044137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 msgr2=0x7fa044137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:43.225+0000 7f337511e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3370138250 0x7f337012ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52328/0 (socket says 172.21.15.139:52328) , 2021-01-15T06:50:43.225+0000 7f337511e700 1 -- 172.21.15.139:0/3793063344 learned_addr learned my addr 172.21.15.139:0/3793063344 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:43.225+0000 7f336ffff700 1 --2- 172.21.15.139:0/3793063344 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f33700a8e20 0x7f337014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:43.400+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f1f24045f50 con 0x7f1f2c0ae1d0 , 2021-01-15T06:50:43.540+0000 7f1f313a4700 1 -- 172.21.15.139:0/2385702852 --> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f2c05d9a0 con 0x7f1f10058210 , 2021-01-15T06:50:43.541+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mgr.4101 v2:172.21.15.139:6832/32872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f2c05d9a0 con 0x7f1f10058210 Ignoring malparsed backtrace: 2021-01-15T06:50:43.825+0000 7f42dad20700 1 -- start start , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42d40683a0 0x7f42d40a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42d40ae1d0 0x7f42d40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 --2- 172.21.15.139:0/2120096777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8dc0683a0 0x7ff8dc0a8e20 unknown :-1 s=CLOSED pgs=4451 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 >> 172.21.15.139:0/2120096777 conn(0x7ff8dc080550 msgr2=0x7ff8dc067ad0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:44.110+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffafc0a67a0 0x7ffafc1319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52420/0 (socket says 172.21.15.139:52420) , 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 -- 172.21.15.139:0/175652128 learned_addr learned my addr 172.21.15.139:0/175652128 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:44.268+0000 7ffb008dc700 1 --2- 172.21.15.139:0/175652128 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffafc0a9b00 0x7ffafc148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800b950 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698025440 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6980257e0 con 0x7fa6b014ac80 Ignoring malparsed backtrace: 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f1e48058210 0x7f1e4805a6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e680ae680 0x7f1e6814e390 unknown :-1 s=CLOSED pgs=4458 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e680683a0 0x7f1e68142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 >> 172.21.15.139:0/3407257751 conn(0x7f312c080550 msgr2=0x7f312c132c70 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 shutdown_connections , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 wait complete. Ignoring malparsed backtrace: 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0009990 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009d10 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c6b2b700 1 -- 172.21.15.139:0/1779367271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62c00a9770 msgr2=0x7f62c013e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:45.588+0000 7fe98dffb700 1 -- 172.21.15.139:0/798497067 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005620 con 0x7fe97c003760 , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 -- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 msgr2=0x7fe97c003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 --2- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 0x7fe97c003b80 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7fe98000c440 tx=0x7fe99013b1e0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0007180 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77bffff700 1 -- 172.21.15.139:0/3931649689 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c00620e0 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0006520 con 0x7f77c00681a0 Ignoring malparsed backtrace: 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 Processor -- start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 -- start start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad2c004a00 0x7fad480ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.503+0000 7fc00497a700 1 -- 172.21.15.139:0/4006517727 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00005d9a0 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 -- 172.21.15.139:0/4006517727 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 5) v1 ==== 68683+0+0 (secure 0 0 0) 0x7fbff4013480 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 --2- 172.21.15.139:0/4006517727 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7fbfdc058210 0x7fbfdc05a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.998+0000 7fad9937b700 1 -- 172.21.15.139:0/1227730117 wait complete. , dumped fsmap epoch 5 , 2021-01-15T06:51:07.013 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json Ignoring malparsed backtrace: 2021-01-15T06:51:07.446+0000 7fb56d59a700 1 -- 172.21.15.139:0/3234842163 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568011610 con 0x7fb5700680b0 , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 -- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 msgr2=0x7fb5700a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 --2- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 0x7fb5700a4ce0 secure :-1 s=READY pgs=4481 cs=0 l=1 rev1=1 rx=0x7fb5700ae0f0 tx=0x7fb5680092d0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f40683a0 0x7fb0f4131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 -- 172.21.15.139:0/1561132054 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4130250 con 0x7fb0f40a8e20 , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0f40a8e20 0x7fb0f4131400 secure :-1 s=READY pgs=4484 cs=0 l=1 rev1=1 rx=0x7fb0e800ebd0 tx=0x7fb0e8006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 -- 172.21.15.139:0/1674202383 shutdown_connections , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f8cf8058240 0x7f8cf805a700 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d14137930 0x7f8d1409daf0 unknown :-1 s=CLOSED pgs=4487 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000bc80 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f84f000bde0 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011650 con 0x7f84f813eed0 Ignoring malparsed backtrace: 2021-01-15T06:51:12.618 INFO:tasks.ceph.mon.a:Stopped , 2021-01-15T06:51:12.619 DEBUG:tasks.ceph.mon.b:waiting for process to exit , 2021-01-15T06:51:12.619 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-15T06:51:33.565 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log , 2021-01-15T06:51:33.620 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-15T06:51:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f36d82f814a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f36d82f8364] Ignoring malparsed backtrace: 2021-01-15T06:50:39.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, , 2021-01-15T06:50:39.606 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { , 2021-01-15T06:50:39.607 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1, Ignoring malparsed backtrace: 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 856, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 2, Ignoring malparsed backtrace: 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "finisher-MDSRank": { , 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "queue_len": 0, , 2021-01-15T06:50:39.841 INFO:teuthology.orchestra.run.smithi149.stdout: "complete_latency": { Ignoring malparsed backtrace: 2021-01-15T06:50:39.876 INFO:teuthology.orchestra.run.smithi149.stdout: "evexg": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "evexd": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "segadd": 20, Ignoring malparsed backtrace: 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "avgcount": 0, , 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "sum": 0.000000000, , 2021-01-15T06:50:39.915 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 Ignoring malparsed backtrace: 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_bytes": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_items": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_other_bytes": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "poolstat_resend": 0, , 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_active": 0, , 2021-01-15T06:50:39.998 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_send": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:40.090 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: } , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: }, Ignoring malparsed backtrace: 2021-01-15T06:50:40.408 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":22,"fsid":"992debe4-2ce0-421a-956c-91da6f0d8c35","created":"2021-01-15T06:47:45.889163+0000","modified":"2021-01-15T06:48:03.144743+0000","last_up_change":"2021-01-15T06:47:52.515218+0000","last_in_change":"2021-01-15T06:47:52.515218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-01-15T06:47:49.538942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-01-15T06:47:56.934802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2021-01-15T06:48:00.119154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2021-01-15T06:48:01.115613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fccfea96-d88f-4a39-810b-fdd874508400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":33082},{"type":"v1","addr":"172.21.15.139:6801","nonce":33082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":33082},{"type":"v1","addr":"172.21.15.139:6803","nonce":33082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":33082},{"type":"v1","addr":"172.21.15.139:6807","nonce":33082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":33082},{"type":"v1","addr":"172.21.15.139:6805","nonce":33082}]},"public_addr":"172.21.15.139:6801/33082","cluster_addr":"172.21.15.139:6803/33082","heartbeat_back_addr":"172.21.15.139:6807/33082","heartbeat_front_addr":"172.21.15.139:6805/33082","state":["exists","up"]},{"osd":1,"uuid":"33e3078f-0ed6-4140-ad99-c833950bf2cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":33083},{"type":"v1","addr":"172.21.15.139:6809","nonce":33083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":33083},{"type":"v1","addr":"172.21.15.139:6811","nonce":33083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":33083},{"type":"v1","addr":"172.21.15.139:6815","nonce":33083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":33083},{"type":"v1","addr":"172.21.15.139:6813","nonce":33083}]},"public_addr":"172.21.15.139:6809/33083","cluster_addr":"172.21.15.139:6811/33083","heartbeat_back_addr":"172.21.15.139:6815/33083","heartbeat_front_addr":"172.21.15.139:6813/33083","state":["exists","up"]},{"osd":2,"uuid":"dac8c79b-1334-4cb2-ab53-f73ac75c9f64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":33085},{"type":"v1","addr":"172.21.15.139:6817","nonce":33085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":33085},{"type":"v1","addr":"172.21.15.139:6819","nonce":33085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":33085},{"type":"v1","addr":"172.21.15.139:6823","nonce":33085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":33085},{"type":"v1","addr":"172.21.15.139:6821","nonce":33085}]},"public_addr":"172.21.15.139:6817/33085","cluster_addr":"172.21.15.139:6819/33085","heartbeat_back_addr":"172.21.15.139:6823/33085","heartbeat_front_addr":"172.21.15.139:6821/33085","state":["exists","up"]},{"osd":3,"uuid":"fc707574-d86e-4c11-afbf-c33d65bd9313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":33123},{"type":"v1","addr":"172.21.15.139:6825","nonce":33123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":33123},{"type":"v1","addr":"172.21.15.139:6827","nonce":33123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":33123},{"type":"v1","addr":"172.21.15.139:6831","nonce":33123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":33123},{"type":"v1","addr":"172.21.15.139:6829","nonce":33123}]},"public_addr":"172.21.15.139:6825/33123","cluster_addr":"172.21.15.139:6827/33123","heartbeat_back_addr":"172.21.15.139:6831/33123","heartbeat_front_addr":"172.21.15.139:6829/33123","state":["exists","up"]},{"osd":4,"uuid":"9bb8a7e3-bec0-4139-909f-91aa80808805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":33318},{"type":"v1","addr":"172.21.15.149:6817","nonce":33318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":33318},{"type":"v1","addr":"172.21.15.149:6823","nonce":33318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6829","nonce":33318},{"type":"v1","addr":"172.21.15.149:6831","nonce":33318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6825","nonce":33318},{"type":"v1","addr":"172.21.15.149:6827","nonce":33318}]},"public_addr":"172.21.15.149:6817/33318","cluster_addr":"172.21.15.149:6823/33318","heartbeat_back_addr":"172.21.15.149:6831/33318","heartbeat_front_addr":"172.21.15.149:6827/33318","state":["exists","up"]},{"osd":5,"uuid":"92842e15-59ef-4944-bea3-46ad21e8a535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":33309},{"type":"v1","addr":"172.21.15.149:6801","nonce":33309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":33309},{"type":"v1","addr":"172.21.15.149:6803","nonce":33309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":33309},{"type":"v1","addr":"172.21.15.149:6807","nonce":33309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":33309},{"type":"v1","addr":"172.21.15.149:6805","nonce":33309}]},"public_addr":"172.21.15.149:6801/33309","cluster_addr":"172.21.15.149:6803/33309","heartbeat_back_addr":"172.21.15.149:6807/33309","heartbeat_front_addr":"172.21.15.149:6805/33309","state":["exists","up"]},{"osd":6,"uuid":"5703d9ed-8fba-446c-83b1-df2f986816ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":33311},{"type":"v1","addr":"172.21.15.149:6809","nonce":33311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":33311},{"type":"v1","addr":"172.21.15.149:6811","nonce":33311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":33311},{"type":"v1","addr":"172.21.15.149:6818","nonce":33311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":33311},{"type":"v1","addr":"172.21.15.149:6813","nonce":33311}]},"public_addr":"172.21.15.149:6809/33311","cluster_addr":"172.21.15.149:6811/33311","heartbeat_back_addr":"172.21.15.149:6818/33311","heartbeat_front_addr":"172.21.15.149:6813/33311","state":["exists","up"]},{"osd":7,"uuid":"4e66a90c-1017-4524-a271-d50c1fa836c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":33320},{"type":"v1","addr":"172.21.15.149:6819","nonce":33320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6821","nonce":33320},{"type":"v1","addr":"172.21.15.149:6822","nonce":33320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":33320},{"type":"v1","addr":"172.21.15.149:6830","nonce":33320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":33320},{"type":"v1","addr":"172.21.15.149:6826","nonce":33320}]},"public_addr":"172.21.15.149:6819/33320","cluster_addr":"172.21.15.149:6822/33320","heartbeat_back_addr":"172.21.15.149:6830/33320","heartbeat_front_addr":"172.21.15.149:6826/33320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.429293+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.402855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.428641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.466464+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.430731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.447289+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.511915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.516104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-01-15T06:50:40.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-15T06:50:40.522+0000 7f8ecedcf700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 shutdown_connections , 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 wait complete. , 2021-01-15T06:50:40.805+0000 7f12c18b2700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:42.098+0000 7ff7277fe700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7280683a0 0x7ff728137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.098+0000 7ff726ffd700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7280a8e20 0x7ff728136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.099+0000 7ff727fff700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff728138250 0x7ff72813d910 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7ff72000b750 tx=0x7ff720017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d900683a0 0x7f7d90136d40 secure :-1 s=READY pgs=4437 cs=0 l=1 rev1=1 rx=0x7f7d900b16b0 tx=0x7f7d80016000).stop , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 -- 172.21.15.139:0/2284323119 shutdown_connections , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d900ae680 0x7f7d9014a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 learned_addr learned my addr 172.21.15.139:0/2781215916 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:42.943+0000 7fa04259c700 1 --2- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 0x7fa044137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 msgr2=0x7fa044137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:43.225+0000 7f337511e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3370138250 0x7f337012ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52328/0 (socket says 172.21.15.139:52328) , 2021-01-15T06:50:43.225+0000 7f337511e700 1 -- 172.21.15.139:0/3793063344 learned_addr learned my addr 172.21.15.139:0/3793063344 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:43.225+0000 7f336ffff700 1 --2- 172.21.15.139:0/3793063344 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f33700a8e20 0x7f337014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:43.400+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f1f24045f50 con 0x7f1f2c0ae1d0 , 2021-01-15T06:50:43.540+0000 7f1f313a4700 1 -- 172.21.15.139:0/2385702852 --> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f2c05d9a0 con 0x7f1f10058210 , 2021-01-15T06:50:43.541+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mgr.4101 v2:172.21.15.139:6832/32872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f2c05d9a0 con 0x7f1f10058210 Ignoring malparsed backtrace: 2021-01-15T06:50:43.825+0000 7f42dad20700 1 -- start start , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42d40683a0 0x7f42d40a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42d40ae1d0 0x7f42d40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 --2- 172.21.15.139:0/2120096777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8dc0683a0 0x7ff8dc0a8e20 unknown :-1 s=CLOSED pgs=4451 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 >> 172.21.15.139:0/2120096777 conn(0x7ff8dc080550 msgr2=0x7ff8dc067ad0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:44.110+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffafc0a67a0 0x7ffafc1319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52420/0 (socket says 172.21.15.139:52420) , 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 -- 172.21.15.139:0/175652128 learned_addr learned my addr 172.21.15.139:0/175652128 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:44.268+0000 7ffb008dc700 1 --2- 172.21.15.139:0/175652128 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffafc0a9b00 0x7ffafc148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800b950 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698025440 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6980257e0 con 0x7fa6b014ac80 Ignoring malparsed backtrace: 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f1e48058210 0x7f1e4805a6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e680ae680 0x7f1e6814e390 unknown :-1 s=CLOSED pgs=4458 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e680683a0 0x7f1e68142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 >> 172.21.15.139:0/3407257751 conn(0x7f312c080550 msgr2=0x7f312c132c70 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 shutdown_connections , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 wait complete. Ignoring malparsed backtrace: 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0009990 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009d10 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c6b2b700 1 -- 172.21.15.139:0/1779367271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62c00a9770 msgr2=0x7f62c013e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:45.588+0000 7fe98dffb700 1 -- 172.21.15.139:0/798497067 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005620 con 0x7fe97c003760 , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 -- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 msgr2=0x7fe97c003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 --2- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 0x7fe97c003b80 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7fe98000c440 tx=0x7fe99013b1e0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0007180 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77bffff700 1 -- 172.21.15.139:0/3931649689 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c00620e0 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0006520 con 0x7f77c00681a0 Ignoring malparsed backtrace: 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 Processor -- start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 -- start start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad2c004a00 0x7fad480ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.503+0000 7fc00497a700 1 -- 172.21.15.139:0/4006517727 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00005d9a0 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 -- 172.21.15.139:0/4006517727 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 5) v1 ==== 68683+0+0 (secure 0 0 0) 0x7fbff4013480 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 --2- 172.21.15.139:0/4006517727 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7fbfdc058210 0x7fbfdc05a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.998+0000 7fad9937b700 1 -- 172.21.15.139:0/1227730117 wait complete. , dumped fsmap epoch 5 , 2021-01-15T06:51:07.013 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json Ignoring malparsed backtrace: 2021-01-15T06:51:07.446+0000 7fb56d59a700 1 -- 172.21.15.139:0/3234842163 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568011610 con 0x7fb5700680b0 , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 -- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 msgr2=0x7fb5700a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 --2- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 0x7fb5700a4ce0 secure :-1 s=READY pgs=4481 cs=0 l=1 rev1=1 rx=0x7fb5700ae0f0 tx=0x7fb5680092d0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f40683a0 0x7fb0f4131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 -- 172.21.15.139:0/1561132054 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4130250 con 0x7fb0f40a8e20 , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0f40a8e20 0x7fb0f4131400 secure :-1 s=READY pgs=4484 cs=0 l=1 rev1=1 rx=0x7fb0e800ebd0 tx=0x7fb0e8006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 -- 172.21.15.139:0/1674202383 shutdown_connections , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f8cf8058240 0x7f8cf805a700 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d14137930 0x7f8d1409daf0 unknown :-1 s=CLOSED pgs=4487 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000bc80 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f84f000bde0 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011650 con 0x7f84f813eed0 Ignoring malparsed backtrace: 2021-01-15T06:51:12.618 INFO:tasks.ceph.mon.a:Stopped , 2021-01-15T06:51:12.619 DEBUG:tasks.ceph.mon.b:waiting for process to exit , 2021-01-15T06:51:12.619 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-15T06:51:33.565 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log , 2021-01-15T06:51:33.620 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-15T06:51:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f36d82f814a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f36d82f8364] Ignoring malparsed backtrace: 2021-01-15T06:50:39.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, , 2021-01-15T06:50:39.606 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { , 2021-01-15T06:50:39.607 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1, Ignoring malparsed backtrace: 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 856, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 2, Ignoring malparsed backtrace: 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "finisher-MDSRank": { , 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "queue_len": 0, , 2021-01-15T06:50:39.841 INFO:teuthology.orchestra.run.smithi149.stdout: "complete_latency": { Ignoring malparsed backtrace: 2021-01-15T06:50:39.876 INFO:teuthology.orchestra.run.smithi149.stdout: "evexg": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "evexd": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "segadd": 20, Ignoring malparsed backtrace: 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "avgcount": 0, , 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "sum": 0.000000000, , 2021-01-15T06:50:39.915 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 Ignoring malparsed backtrace: 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_bytes": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_items": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_other_bytes": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "poolstat_resend": 0, , 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_active": 0, , 2021-01-15T06:50:39.998 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_send": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:40.090 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: } , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: }, Ignoring malparsed backtrace: 2021-01-15T06:50:40.408 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":22,"fsid":"992debe4-2ce0-421a-956c-91da6f0d8c35","created":"2021-01-15T06:47:45.889163+0000","modified":"2021-01-15T06:48:03.144743+0000","last_up_change":"2021-01-15T06:47:52.515218+0000","last_in_change":"2021-01-15T06:47:52.515218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-01-15T06:47:49.538942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-01-15T06:47:56.934802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2021-01-15T06:48:00.119154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2021-01-15T06:48:01.115613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fccfea96-d88f-4a39-810b-fdd874508400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":33082},{"type":"v1","addr":"172.21.15.139:6801","nonce":33082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":33082},{"type":"v1","addr":"172.21.15.139:6803","nonce":33082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":33082},{"type":"v1","addr":"172.21.15.139:6807","nonce":33082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":33082},{"type":"v1","addr":"172.21.15.139:6805","nonce":33082}]},"public_addr":"172.21.15.139:6801/33082","cluster_addr":"172.21.15.139:6803/33082","heartbeat_back_addr":"172.21.15.139:6807/33082","heartbeat_front_addr":"172.21.15.139:6805/33082","state":["exists","up"]},{"osd":1,"uuid":"33e3078f-0ed6-4140-ad99-c833950bf2cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":33083},{"type":"v1","addr":"172.21.15.139:6809","nonce":33083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":33083},{"type":"v1","addr":"172.21.15.139:6811","nonce":33083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":33083},{"type":"v1","addr":"172.21.15.139:6815","nonce":33083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":33083},{"type":"v1","addr":"172.21.15.139:6813","nonce":33083}]},"public_addr":"172.21.15.139:6809/33083","cluster_addr":"172.21.15.139:6811/33083","heartbeat_back_addr":"172.21.15.139:6815/33083","heartbeat_front_addr":"172.21.15.139:6813/33083","state":["exists","up"]},{"osd":2,"uuid":"dac8c79b-1334-4cb2-ab53-f73ac75c9f64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":33085},{"type":"v1","addr":"172.21.15.139:6817","nonce":33085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":33085},{"type":"v1","addr":"172.21.15.139:6819","nonce":33085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":33085},{"type":"v1","addr":"172.21.15.139:6823","nonce":33085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":33085},{"type":"v1","addr":"172.21.15.139:6821","nonce":33085}]},"public_addr":"172.21.15.139:6817/33085","cluster_addr":"172.21.15.139:6819/33085","heartbeat_back_addr":"172.21.15.139:6823/33085","heartbeat_front_addr":"172.21.15.139:6821/33085","state":["exists","up"]},{"osd":3,"uuid":"fc707574-d86e-4c11-afbf-c33d65bd9313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":33123},{"type":"v1","addr":"172.21.15.139:6825","nonce":33123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":33123},{"type":"v1","addr":"172.21.15.139:6827","nonce":33123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":33123},{"type":"v1","addr":"172.21.15.139:6831","nonce":33123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":33123},{"type":"v1","addr":"172.21.15.139:6829","nonce":33123}]},"public_addr":"172.21.15.139:6825/33123","cluster_addr":"172.21.15.139:6827/33123","heartbeat_back_addr":"172.21.15.139:6831/33123","heartbeat_front_addr":"172.21.15.139:6829/33123","state":["exists","up"]},{"osd":4,"uuid":"9bb8a7e3-bec0-4139-909f-91aa80808805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":33318},{"type":"v1","addr":"172.21.15.149:6817","nonce":33318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":33318},{"type":"v1","addr":"172.21.15.149:6823","nonce":33318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6829","nonce":33318},{"type":"v1","addr":"172.21.15.149:6831","nonce":33318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6825","nonce":33318},{"type":"v1","addr":"172.21.15.149:6827","nonce":33318}]},"public_addr":"172.21.15.149:6817/33318","cluster_addr":"172.21.15.149:6823/33318","heartbeat_back_addr":"172.21.15.149:6831/33318","heartbeat_front_addr":"172.21.15.149:6827/33318","state":["exists","up"]},{"osd":5,"uuid":"92842e15-59ef-4944-bea3-46ad21e8a535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":33309},{"type":"v1","addr":"172.21.15.149:6801","nonce":33309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":33309},{"type":"v1","addr":"172.21.15.149:6803","nonce":33309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":33309},{"type":"v1","addr":"172.21.15.149:6807","nonce":33309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":33309},{"type":"v1","addr":"172.21.15.149:6805","nonce":33309}]},"public_addr":"172.21.15.149:6801/33309","cluster_addr":"172.21.15.149:6803/33309","heartbeat_back_addr":"172.21.15.149:6807/33309","heartbeat_front_addr":"172.21.15.149:6805/33309","state":["exists","up"]},{"osd":6,"uuid":"5703d9ed-8fba-446c-83b1-df2f986816ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":33311},{"type":"v1","addr":"172.21.15.149:6809","nonce":33311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":33311},{"type":"v1","addr":"172.21.15.149:6811","nonce":33311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":33311},{"type":"v1","addr":"172.21.15.149:6818","nonce":33311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":33311},{"type":"v1","addr":"172.21.15.149:6813","nonce":33311}]},"public_addr":"172.21.15.149:6809/33311","cluster_addr":"172.21.15.149:6811/33311","heartbeat_back_addr":"172.21.15.149:6818/33311","heartbeat_front_addr":"172.21.15.149:6813/33311","state":["exists","up"]},{"osd":7,"uuid":"4e66a90c-1017-4524-a271-d50c1fa836c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":33320},{"type":"v1","addr":"172.21.15.149:6819","nonce":33320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6821","nonce":33320},{"type":"v1","addr":"172.21.15.149:6822","nonce":33320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":33320},{"type":"v1","addr":"172.21.15.149:6830","nonce":33320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":33320},{"type":"v1","addr":"172.21.15.149:6826","nonce":33320}]},"public_addr":"172.21.15.149:6819/33320","cluster_addr":"172.21.15.149:6822/33320","heartbeat_back_addr":"172.21.15.149:6830/33320","heartbeat_front_addr":"172.21.15.149:6826/33320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.429293+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.402855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.428641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.466464+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.430731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.447289+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.511915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.516104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-01-15T06:50:40.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-15T06:50:40.522+0000 7f8ecedcf700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 shutdown_connections , 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 wait complete. , 2021-01-15T06:50:40.805+0000 7f12c18b2700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:42.098+0000 7ff7277fe700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7280683a0 0x7ff728137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.098+0000 7ff726ffd700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7280a8e20 0x7ff728136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.099+0000 7ff727fff700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff728138250 0x7ff72813d910 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7ff72000b750 tx=0x7ff720017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d900683a0 0x7f7d90136d40 secure :-1 s=READY pgs=4437 cs=0 l=1 rev1=1 rx=0x7f7d900b16b0 tx=0x7f7d80016000).stop , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 -- 172.21.15.139:0/2284323119 shutdown_connections , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d900ae680 0x7f7d9014a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 learned_addr learned my addr 172.21.15.139:0/2781215916 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:42.943+0000 7fa04259c700 1 --2- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 0x7fa044137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 msgr2=0x7fa044137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:43.225+0000 7f337511e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3370138250 0x7f337012ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52328/0 (socket says 172.21.15.139:52328) , 2021-01-15T06:50:43.225+0000 7f337511e700 1 -- 172.21.15.139:0/3793063344 learned_addr learned my addr 172.21.15.139:0/3793063344 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:43.225+0000 7f336ffff700 1 --2- 172.21.15.139:0/3793063344 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f33700a8e20 0x7f337014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:43.400+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f1f24045f50 con 0x7f1f2c0ae1d0 , 2021-01-15T06:50:43.540+0000 7f1f313a4700 1 -- 172.21.15.139:0/2385702852 --> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f2c05d9a0 con 0x7f1f10058210 , 2021-01-15T06:50:43.541+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mgr.4101 v2:172.21.15.139:6832/32872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f2c05d9a0 con 0x7f1f10058210 Ignoring malparsed backtrace: 2021-01-15T06:50:43.825+0000 7f42dad20700 1 -- start start , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42d40683a0 0x7f42d40a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42d40ae1d0 0x7f42d40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 --2- 172.21.15.139:0/2120096777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8dc0683a0 0x7ff8dc0a8e20 unknown :-1 s=CLOSED pgs=4451 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 >> 172.21.15.139:0/2120096777 conn(0x7ff8dc080550 msgr2=0x7ff8dc067ad0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:44.110+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffafc0a67a0 0x7ffafc1319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52420/0 (socket says 172.21.15.139:52420) , 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 -- 172.21.15.139:0/175652128 learned_addr learned my addr 172.21.15.139:0/175652128 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:44.268+0000 7ffb008dc700 1 --2- 172.21.15.139:0/175652128 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffafc0a9b00 0x7ffafc148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800b950 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698025440 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6980257e0 con 0x7fa6b014ac80 Ignoring malparsed backtrace: 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f1e48058210 0x7f1e4805a6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e680ae680 0x7f1e6814e390 unknown :-1 s=CLOSED pgs=4458 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e680683a0 0x7f1e68142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 >> 172.21.15.139:0/3407257751 conn(0x7f312c080550 msgr2=0x7f312c132c70 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 shutdown_connections , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 wait complete. Ignoring malparsed backtrace: 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0009990 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009d10 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c6b2b700 1 -- 172.21.15.139:0/1779367271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62c00a9770 msgr2=0x7f62c013e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:45.588+0000 7fe98dffb700 1 -- 172.21.15.139:0/798497067 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005620 con 0x7fe97c003760 , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 -- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 msgr2=0x7fe97c003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 --2- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 0x7fe97c003b80 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7fe98000c440 tx=0x7fe99013b1e0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0007180 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77bffff700 1 -- 172.21.15.139:0/3931649689 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c00620e0 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0006520 con 0x7f77c00681a0 Ignoring malparsed backtrace: 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 Processor -- start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 -- start start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad2c004a00 0x7fad480ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.503+0000 7fc00497a700 1 -- 172.21.15.139:0/4006517727 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00005d9a0 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 -- 172.21.15.139:0/4006517727 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 5) v1 ==== 68683+0+0 (secure 0 0 0) 0x7fbff4013480 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 --2- 172.21.15.139:0/4006517727 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7fbfdc058210 0x7fbfdc05a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.998+0000 7fad9937b700 1 -- 172.21.15.139:0/1227730117 wait complete. , dumped fsmap epoch 5 , 2021-01-15T06:51:07.013 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json Ignoring malparsed backtrace: 2021-01-15T06:51:07.446+0000 7fb56d59a700 1 -- 172.21.15.139:0/3234842163 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568011610 con 0x7fb5700680b0 , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 -- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 msgr2=0x7fb5700a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 --2- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 0x7fb5700a4ce0 secure :-1 s=READY pgs=4481 cs=0 l=1 rev1=1 rx=0x7fb5700ae0f0 tx=0x7fb5680092d0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f40683a0 0x7fb0f4131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 -- 172.21.15.139:0/1561132054 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4130250 con 0x7fb0f40a8e20 , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0f40a8e20 0x7fb0f4131400 secure :-1 s=READY pgs=4484 cs=0 l=1 rev1=1 rx=0x7fb0e800ebd0 tx=0x7fb0e8006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 -- 172.21.15.139:0/1674202383 shutdown_connections , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f8cf8058240 0x7f8cf805a700 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d14137930 0x7f8d1409daf0 unknown :-1 s=CLOSED pgs=4487 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000bc80 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f84f000bde0 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011650 con 0x7f84f813eed0 Ignoring malparsed backtrace: 2021-01-15T06:51:12.618 INFO:tasks.ceph.mon.a:Stopped , 2021-01-15T06:51:12.619 DEBUG:tasks.ceph.mon.b:waiting for process to exit , 2021-01-15T06:51:12.619 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-15T06:51:33.565 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log , 2021-01-15T06:51:33.620 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-15T06:51:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 Ignoring malparsed backtrace: ceph version 16.0.0-9044-g9079b095 (9079b095d00da9a7a18a07fca429d8b555325955) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f36d82f814a] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276364) [0x7f36d82f8364] Ignoring malparsed backtrace: 2021-01-15T06:50:39.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, , 2021-01-15T06:50:39.606 INFO:teuthology.orchestra.run.smithi139.stdout: "AsyncMessenger::Worker-1": { , 2021-01-15T06:50:39.607 INFO:teuthology.orchestra.run.smithi139.stdout: "msgr_recv_messages": 1, Ignoring malparsed backtrace: 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "pgmap_items": 0, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_bytes": 856, , 2021-01-15T06:50:39.651 INFO:teuthology.orchestra.run.smithi139.stdout: "mds_co_items": 2, Ignoring malparsed backtrace: 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "finisher-MDSRank": { , 2021-01-15T06:50:39.840 INFO:teuthology.orchestra.run.smithi149.stdout: "queue_len": 0, , 2021-01-15T06:50:39.841 INFO:teuthology.orchestra.run.smithi149.stdout: "complete_latency": { Ignoring malparsed backtrace: 2021-01-15T06:50:39.876 INFO:teuthology.orchestra.run.smithi149.stdout: "evexg": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "evexd": 0, , 2021-01-15T06:50:39.877 INFO:teuthology.orchestra.run.smithi149.stdout: "segadd": 20, Ignoring malparsed backtrace: 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "avgcount": 0, , 2021-01-15T06:50:39.914 INFO:teuthology.orchestra.run.smithi149.stdout: "sum": 0.000000000, , 2021-01-15T06:50:39.915 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 Ignoring malparsed backtrace: 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_bytes": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_meta_items": 0, , 2021-01-15T06:50:39.952 INFO:teuthology.orchestra.run.smithi149.stdout: "bluestore_cache_other_bytes": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "poolstat_resend": 0, , 2021-01-15T06:50:39.997 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_active": 0, , 2021-01-15T06:50:39.998 INFO:teuthology.orchestra.run.smithi149.stdout: "statfs_send": 0, Ignoring malparsed backtrace: 2021-01-15T06:50:40.090 INFO:teuthology.orchestra.run.smithi149.stdout: "avgtime": 0.000000000 , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: } , 2021-01-15T06:50:40.091 INFO:teuthology.orchestra.run.smithi149.stdout: }, Ignoring malparsed backtrace: 2021-01-15T06:50:40.408 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":22,"fsid":"992debe4-2ce0-421a-956c-91da6f0d8c35","created":"2021-01-15T06:47:45.889163+0000","modified":"2021-01-15T06:48:03.144743+0000","last_up_change":"2021-01-15T06:47:52.515218+0000","last_in_change":"2021-01-15T06:47:52.515218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-01-15T06:47:49.538942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-01-15T06:47:56.934802+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2021-01-15T06:48:00.119154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2021-01-15T06:48:01.115613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fccfea96-d88f-4a39-810b-fdd874508400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":33082},{"type":"v1","addr":"172.21.15.139:6801","nonce":33082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":33082},{"type":"v1","addr":"172.21.15.139:6803","nonce":33082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":33082},{"type":"v1","addr":"172.21.15.139:6807","nonce":33082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":33082},{"type":"v1","addr":"172.21.15.139:6805","nonce":33082}]},"public_addr":"172.21.15.139:6801/33082","cluster_addr":"172.21.15.139:6803/33082","heartbeat_back_addr":"172.21.15.139:6807/33082","heartbeat_front_addr":"172.21.15.139:6805/33082","state":["exists","up"]},{"osd":1,"uuid":"33e3078f-0ed6-4140-ad99-c833950bf2cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":33083},{"type":"v1","addr":"172.21.15.139:6809","nonce":33083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":33083},{"type":"v1","addr":"172.21.15.139:6811","nonce":33083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":33083},{"type":"v1","addr":"172.21.15.139:6815","nonce":33083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":33083},{"type":"v1","addr":"172.21.15.139:6813","nonce":33083}]},"public_addr":"172.21.15.139:6809/33083","cluster_addr":"172.21.15.139:6811/33083","heartbeat_back_addr":"172.21.15.139:6815/33083","heartbeat_front_addr":"172.21.15.139:6813/33083","state":["exists","up"]},{"osd":2,"uuid":"dac8c79b-1334-4cb2-ab53-f73ac75c9f64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":33085},{"type":"v1","addr":"172.21.15.139:6817","nonce":33085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":33085},{"type":"v1","addr":"172.21.15.139:6819","nonce":33085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":33085},{"type":"v1","addr":"172.21.15.139:6823","nonce":33085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":33085},{"type":"v1","addr":"172.21.15.139:6821","nonce":33085}]},"public_addr":"172.21.15.139:6817/33085","cluster_addr":"172.21.15.139:6819/33085","heartbeat_back_addr":"172.21.15.139:6823/33085","heartbeat_front_addr":"172.21.15.139:6821/33085","state":["exists","up"]},{"osd":3,"uuid":"fc707574-d86e-4c11-afbf-c33d65bd9313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":33123},{"type":"v1","addr":"172.21.15.139:6825","nonce":33123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":33123},{"type":"v1","addr":"172.21.15.139:6827","nonce":33123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":33123},{"type":"v1","addr":"172.21.15.139:6831","nonce":33123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":33123},{"type":"v1","addr":"172.21.15.139:6829","nonce":33123}]},"public_addr":"172.21.15.139:6825/33123","cluster_addr":"172.21.15.139:6827/33123","heartbeat_back_addr":"172.21.15.139:6831/33123","heartbeat_front_addr":"172.21.15.139:6829/33123","state":["exists","up"]},{"osd":4,"uuid":"9bb8a7e3-bec0-4139-909f-91aa80808805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":33318},{"type":"v1","addr":"172.21.15.149:6817","nonce":33318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":33318},{"type":"v1","addr":"172.21.15.149:6823","nonce":33318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6829","nonce":33318},{"type":"v1","addr":"172.21.15.149:6831","nonce":33318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6825","nonce":33318},{"type":"v1","addr":"172.21.15.149:6827","nonce":33318}]},"public_addr":"172.21.15.149:6817/33318","cluster_addr":"172.21.15.149:6823/33318","heartbeat_back_addr":"172.21.15.149:6831/33318","heartbeat_front_addr":"172.21.15.149:6827/33318","state":["exists","up"]},{"osd":5,"uuid":"92842e15-59ef-4944-bea3-46ad21e8a535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":33309},{"type":"v1","addr":"172.21.15.149:6801","nonce":33309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":33309},{"type":"v1","addr":"172.21.15.149:6803","nonce":33309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":33309},{"type":"v1","addr":"172.21.15.149:6807","nonce":33309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":33309},{"type":"v1","addr":"172.21.15.149:6805","nonce":33309}]},"public_addr":"172.21.15.149:6801/33309","cluster_addr":"172.21.15.149:6803/33309","heartbeat_back_addr":"172.21.15.149:6807/33309","heartbeat_front_addr":"172.21.15.149:6805/33309","state":["exists","up"]},{"osd":6,"uuid":"5703d9ed-8fba-446c-83b1-df2f986816ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":33311},{"type":"v1","addr":"172.21.15.149:6809","nonce":33311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":33311},{"type":"v1","addr":"172.21.15.149:6811","nonce":33311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6815","nonce":33311},{"type":"v1","addr":"172.21.15.149:6818","nonce":33311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":33311},{"type":"v1","addr":"172.21.15.149:6813","nonce":33311}]},"public_addr":"172.21.15.149:6809/33311","cluster_addr":"172.21.15.149:6811/33311","heartbeat_back_addr":"172.21.15.149:6818/33311","heartbeat_front_addr":"172.21.15.149:6813/33311","state":["exists","up"]},{"osd":7,"uuid":"4e66a90c-1017-4524-a271-d50c1fa836c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":33320},{"type":"v1","addr":"172.21.15.149:6819","nonce":33320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6821","nonce":33320},{"type":"v1","addr":"172.21.15.149:6822","nonce":33320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":33320},{"type":"v1","addr":"172.21.15.149:6830","nonce":33320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":33320},{"type":"v1","addr":"172.21.15.149:6826","nonce":33320}]},"public_addr":"172.21.15.149:6819/33320","cluster_addr":"172.21.15.149:6822/33320","heartbeat_back_addr":"172.21.15.149:6830/33320","heartbeat_front_addr":"172.21.15.149:6826/33320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.429293+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.402855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.428641+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.466464+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.430731+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.447289+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.511915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-01-15T06:47:51.516104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-01-15T06:50:40.411 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-15T06:50:40.522+0000 7f8ecedcf700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 shutdown_connections , 2021-01-15T06:50:40.804+0000 7f12c18b2700 1 -- 172.21.15.139:0/506978489 wait complete. , 2021-01-15T06:50:40.805+0000 7f12c18b2700 1 Processor -- start Ignoring malparsed backtrace: 2021-01-15T06:50:42.098+0000 7ff7277fe700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7280683a0 0x7ff728137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.098+0000 7ff726ffd700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7280a8e20 0x7ff728136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! , 2021-01-15T06:50:42.099+0000 7ff727fff700 1 --2- 172.21.15.139:0/3161270281 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff728138250 0x7ff72813d910 secure :-1 s=READY pgs=4435 cs=0 l=1 rev1=1 rx=0x7ff72000b750 tx=0x7ff720017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7d900683a0 0x7f7d90136d40 secure :-1 s=READY pgs=4437 cs=0 l=1 rev1=1 rx=0x7f7d900b16b0 tx=0x7f7d80016000).stop , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 -- 172.21.15.139:0/2284323119 shutdown_connections , 2021-01-15T06:50:42.642+0000 7f7d94f99700 1 --2- 172.21.15.139:0/2284323119 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d900ae680 0x7f7d9014a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 learned_addr learned my addr 172.21.15.139:0/2781215916 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:42.943+0000 7fa04259c700 1 --2- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 0x7fa044137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-01-15T06:50:42.943+0000 7fa04359e700 1 -- 172.21.15.139:0/2781215916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa0440a8e20 msgr2=0x7fa044137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:43.225+0000 7f337511e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3370138250 0x7f337012ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52328/0 (socket says 172.21.15.139:52328) , 2021-01-15T06:50:43.225+0000 7f337511e700 1 -- 172.21.15.139:0/3793063344 learned_addr learned my addr 172.21.15.139:0/3793063344 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:43.225+0000 7f336ffff700 1 --2- 172.21.15.139:0/3793063344 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f33700a8e20 0x7f337014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:43.400+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+144453 (secure 0 0 0) 0x7f1f24045f50 con 0x7f1f2c0ae1d0 , 2021-01-15T06:50:43.540+0000 7f1f313a4700 1 -- 172.21.15.139:0/2385702852 --> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f2c05d9a0 con 0x7f1f10058210 , 2021-01-15T06:50:43.541+0000 7f1f22ffd700 1 -- 172.21.15.139:0/2385702852 <== mgr.4101 v2:172.21.15.139:6832/32872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f2c05d9a0 con 0x7f1f10058210 Ignoring malparsed backtrace: 2021-01-15T06:50:43.825+0000 7f42dad20700 1 -- start start , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42d40683a0 0x7f42d40a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-01-15T06:50:43.826+0000 7f42dad20700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42d40ae1d0 0x7f42d40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 --2- 172.21.15.139:0/2120096777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff8dc0683a0 0x7ff8dc0a8e20 unknown :-1 s=CLOSED pgs=4451 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.109+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 >> 172.21.15.139:0/2120096777 conn(0x7ff8dc080550 msgr2=0x7ff8dc067ad0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:44.110+0000 7ff8e29be700 1 -- 172.21.15.139:0/2120096777 shutdown_connections Ignoring malparsed backtrace: 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffafc0a67a0 0x7ffafc1319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:52420/0 (socket says 172.21.15.139:52420) , 2021-01-15T06:50:44.268+0000 7ffb010dd700 1 -- 172.21.15.139:0/175652128 learned_addr learned my addr 172.21.15.139:0/175652128 (peer_addr_for_me v2:172.21.15.139:0/0) , 2021-01-15T06:50:44.268+0000 7ffb008dc700 1 --2- 172.21.15.139:0/175652128 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffafc0a9b00 0x7ffafc148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800b950 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698025440 con 0x7fa6b014ac80 , 2021-01-15T06:50:44.546+0000 7fa6a7fff700 1 -- 172.21.15.139:0/1160664256 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6980257e0 con 0x7fa6b014ac80 Ignoring malparsed backtrace: 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f1e48058210 0x7f1e4805a6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1e680ae680 0x7f1e6814e390 unknown :-1 s=CLOSED pgs=4458 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:50:44.856+0000 7f1e6eab4700 1 --2- 172.21.15.139:0/1350601675 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e680683a0 0x7f1e68142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 >> 172.21.15.139:0/3407257751 conn(0x7f312c080550 msgr2=0x7f312c132c70 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 shutdown_connections , 2021-01-15T06:50:45.004+0000 7f3130c18700 1 -- 172.21.15.139:0/3407257751 wait complete. Ignoring malparsed backtrace: 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0009990 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c4b27700 1 -- 172.21.15.139:0/1779367271 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009d10 con 0x7f62c00a9770 , 2021-01-15T06:50:45.428+0000 7f62c6b2b700 1 -- 172.21.15.139:0/1779367271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f62c00a9770 msgr2=0x7f62c013e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-01-15T06:50:45.588+0000 7fe98dffb700 1 -- 172.21.15.139:0/798497067 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005620 con 0x7fe97c003760 , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 -- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 msgr2=0x7fe97c003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:50:45.589+0000 7fe995a59700 1 --2- 172.21.15.139:0/798497067 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c003760 0x7fe97c003b80 secure :-1 s=READY pgs=4468 cs=0 l=1 rev1=1 rx=0x7fe98000c440 tx=0x7fe99013b1e0).stop Ignoring malparsed backtrace: 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0007180 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77bffff700 1 -- 172.21.15.139:0/3931649689 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c00620e0 con 0x7f77c00681a0 , 2021-01-15T06:50:45.883+0000 7f77a6ffd700 1 -- 172.21.15.139:0/3931649689 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0006520 con 0x7f77c00681a0 Ignoring malparsed backtrace: 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 Processor -- start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 -- start start , 2021-01-15T06:51:06.214+0000 7fad4cf0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad2c004a00 0x7fad480ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.503+0000 7fc00497a700 1 -- 172.21.15.139:0/4006517727 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00005d9a0 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 -- 172.21.15.139:0/4006517727 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 5) v1 ==== 68683+0+0 (secure 0 0 0) 0x7fbff4013480 con 0x7fc000138370 , 2021-01-15T06:51:06.503+0000 7fbfee7fc700 1 --2- 172.21.15.139:0/4006517727 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7fbfdc058210 0x7fbfdc05a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-01-15T06:51:06.998+0000 7fad9937b700 1 -- 172.21.15.139:0/1227730117 wait complete. , dumped fsmap epoch 5 , 2021-01-15T06:51:07.013 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json Ignoring malparsed backtrace: 2021-01-15T06:51:07.446+0000 7fb56d59a700 1 -- 172.21.15.139:0/3234842163 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568011610 con 0x7fb5700680b0 , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 -- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 msgr2=0x7fb5700a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-01-15T06:51:07.446+0000 7fb56f59e700 1 --2- 172.21.15.139:0/3234842163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5700680b0 0x7fb5700a4ce0 secure :-1 s=READY pgs=4481 cs=0 l=1 rev1=1 rx=0x7fb5700ae0f0 tx=0x7fb5680092d0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb0f40683a0 0x7fb0f4131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 -- 172.21.15.139:0/1561132054 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4130250 con 0x7fb0f40a8e20 , 2021-01-15T06:51:07.733+0000 7fb0f37fe700 1 --2- 172.21.15.139:0/1561132054 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0f40a8e20 0x7fb0f4131400 secure :-1 s=READY pgs=4484 cs=0 l=1 rev1=1 rx=0x7fb0e800ebd0 tx=0x7fb0e8006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 -- 172.21.15.139:0/1674202383 shutdown_connections , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:6832/32872,v1:172.21.15.139:6833/32872] conn(0x7f8cf8058240 0x7f8cf805a700 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-01-15T06:51:09.936+0000 7f8d18efa700 1 --2- 172.21.15.139:0/1674202383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d14137930 0x7f8d1409daf0 unknown :-1 s=CLOSED pgs=4487 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f000bc80 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f84f000bde0 con 0x7f84f813eed0 , 2021-01-15T06:51:11.127+0000 7f84f67fc700 1 -- 172.21.15.139:0/2980487064 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011650 con 0x7f84f813eed0 Ignoring malparsed backtrace: 2021-01-15T06:51:12.618 INFO:tasks.ceph.mon.a:Stopped , 2021-01-15T06:51:12.619 DEBUG:tasks.ceph.mon.b:waiting for process to exit , 2021-01-15T06:51:12.619 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-15T06:51:33.565 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log , 2021-01-15T06:51:33.620 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2021-01-15T06:51:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 Found 3 distinct failure reasons Failure: Command failed (workunit test client/test.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9079b095d00da9a7a18a07fca429d8b555325955 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/client/test.sh' 1 jobs: ['5788315'] suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs:libcephfs/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/client}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test fs/test_python.sh) on smithi199 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9079b095d00da9a7a18a07fca429d8b555325955 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 1 jobs: ['5788317'] suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs:libcephfs/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/libcephfs_python}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed (workunit test libcephfs/test.sh) on smithi139 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9079b095d00da9a7a18a07fca429d8b555325955 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 1 jobs: ['5788316'] suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs:libcephfs/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/libcephfs}', 'whitelist_health', 'whitelist_wrongly_marked_down}']