Found 78 jobs Ignoring malparsed backtrace: ceph version 17.0.0-2245-ge841aa06 (e841aa06e2fa775b691b12cf67756986cdba1952) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x558512e59504] , 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x4c9) [0x5585135bd549] Ignoring malparsed backtrace: 2021-03-30T16:51:22.765+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:51:23.754+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:51:24.748+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:53:33.635+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:53:34.605+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:53:35.577+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:55:14.667+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:55:15.672+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:55:16.631+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:56:55.444+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:56:56.490+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:56:57.498+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 28 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:58:36.498+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:58:37.547+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) , 2021-03-30T16:58:38.553+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:00:17.642+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:00:18.671+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:00:19.665+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:01:58.818+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:01:59.830+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:02:00.852+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:03:39.886+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:03:40.919+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:03:41.930+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:05:20.732+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:05:21.698+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:05:22.650+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 51 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:07:01.683+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:07:02.681+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:07:03.687+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:08:43.185+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:08:44.217+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:08:45.235+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:10:24.535+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:10:25.500+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:10:26.476+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:12:04.972+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:12:06.010+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:12:06.978+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:13:46.152+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:13:47.120+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:13:48.127+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:15:27.577+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:15:28.613+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:15:29.569+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:17:08.494+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:17:09.457+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:17:10.418+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:18:49.554+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:18:50.595+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:18:51.554+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:20:31.033+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:20:32.027+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:20:33.046+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:22:11.592+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:22:12.600+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:22:13.616+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 76 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:23:52.798+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:23:53.829+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:23:54.853+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:25:34.283+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:25:35.321+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:25:36.292+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:27:17.161+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:27:18.144+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:27:19.180+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:29:42.863+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 94 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:29:44.336+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 94 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:29:45.366+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 94 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:31:25.504+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:31:26.503+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:31:27.536+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:33:08.802+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:33:09.835+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:33:11.111+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:39:20.833+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:39:37.239+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:39:55.083+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 157 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T18:47:55.413+0000 7f530d175700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530009b370 con 0x7f53000ac050 , 2021-03-30T18:47:55.414+0000 7f5307fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53000ac050 0x7f53000b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:47:55.414+0000 7f5307fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53000ac050 0x7f53000b2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50908/0 (socket says 172.21.15.16:50908) Ignoring malparsed backtrace: 2021-03-30T18:47:56.093+0000 7fa1f67fc700 1 -- 172.21.15.16:0/3471896784 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522657018 0 0) 0x7fa1f808d1c0 con 0x7fa1f811b150 , 2021-03-30T18:47:56.093+0000 7fa1f6ffd700 1 -- 172.21.15.16:0/3471896784 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f811acb0 con 0x7fa1f811dfb0 , 2021-03-30T18:47:56.093+0000 7fa1f6ffd700 1 --2- 172.21.15.16:0/3471896784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f811dfb0 0x7fa1f81203a0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7fa1ec00b940 tx=0x7fa1ec006e20).ready entity=mon.0 client_cookie=89e6bcd86f57b914 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:48:16.599+0000 7f99d1027700 1 --2- 172.21.15.16:0/2825235693 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f99b002f2b0 0x7f99b0031770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:48:16.599+0000 7f99d1027700 1 --2- 172.21.15.16:0/2825235693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99cc088630 0x7f99cc12d150 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:48:16.599+0000 7f99d1027700 1 -- 172.21.15.16:0/2825235693 >> 172.21.15.16:0/2825235693 conn(0x7f99cc08b660 msgr2=0x7f99cc066a00 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:48:37.093+0000 7f81f6ffd700 1 -- 172.21.15.16:0/1493163775 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7f81f000f070 con 0x7f8200089850 , 2021-03-30T18:48:37.093+0000 7f81fffff700 1 -- 172.21.15.16:0/1493163775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82001188e0 con 0x7f8200089850 , 2021-03-30T18:48:37.094+0000 7f81f6ffd700 1 --2- 172.21.15.16:0/1493163775 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f81e402f2b0 0x7f81e4031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-03-30T18:48:57.763+0000 7fa6c0943700 1 -- 172.21.15.16:0/286481548 >> 172.21.15.16:0/286481548 conn(0x7fa6bc062cf0 msgr2=0x7fa6bc065110 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:48:57.763+0000 7fa6c0943700 1 -- 172.21.15.16:0/286481548 shutdown_connections , 2021-03-30T18:48:57.764+0000 7fa6c0943700 1 -- 172.21.15.16:0/286481548 wait complete. Ignoring malparsed backtrace: 2021-03-30T18:49:18.425+0000 7faf20e46700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1c08f380 con 0x7faf1c066650 , 2021-03-30T18:49:18.425+0000 7faf20e46700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1c068660 con 0x7faf1c121b20 , 2021-03-30T18:49:18.425+0000 7faf1affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf1c121b20 0x7faf1c123f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-03-30T18:49:18.948 INFO:teuthology.orchestra.run.smithi016.stdout: , 2021-03-30T18:49:18.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"0b4129c0-5322-4cef-8ab1-aa1d5b1c34aa","created":"2021-03-30T15:44:37.569199+0000","modified":"2021-03-30T18:02:50.582257+0000","last_up_change":"2021-03-30T18:02:50.582257+0000","last_in_change":"2021-03-30T16:41:52.416862+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-30T15:44:43.950286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-30T15:44:46.057469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-03-30T15:44:55.025694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-03-30T15:44:57.303437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-03-30T15:44:59.165384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-03-30T15:45:01.036872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-03-30T15:45:03.014361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-03-30T15:45:05.275172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.buckets.non-ec","create_time":"2021-03-30T15:46:14.040116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4f3b9446-e5a3-4faf-b50f-0fbc3c82709a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":33,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":27827},{"type":"v1","addr":"172.21.15.16:6805","nonce":27827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":27827},{"type":"v1","addr":"172.21.15.16:6809","nonce":27827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":27827},{"type":"v1","addr":"172.21.15.16:6815","nonce":27827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":27827},{"type":"v1","addr":"172.21.15.16:6813","nonce":27827}]},"public_addr":"172.21.15.16:6805/27827","cluster_addr":"172.21.15.16:6809/27827","heartbeat_back_addr":"172.21.15.16:6815/27827","heartbeat_front_addr":"172.21.15.16:6813/27827","state":["autoout","exists"]},{"osd":1,"uuid":"3f7e8b67-ed65-4e68-841a-9cc5b6176b97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":38,"up_from":39,"up_thru":39,"down_at":41,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":27828},{"type":"v1","addr":"172.21.15.16:6801","nonce":27828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2027828},{"type":"v1","addr":"172.21.15.16:6804","nonce":2027828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":27828},{"type":"v1","addr":"172.21.15.16:6812","nonce":27828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":27828},{"type":"v1","addr":"172.21.15.16:6808","nonce":27828}]},"public_addr":"172.21.15.16:6801/27828","cluster_addr":"172.21.15.16:6804/2027828","heartbeat_back_addr":"172.21.15.16:6812/27828","heartbeat_front_addr":"172.21.15.16:6808/27828","state":["exists"]},{"osd":2,"uuid":"6cfa9335-1f9b-4d97-b631-847ef2077610","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":33,"up_from":34,"up_thru":34,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":27851},{"type":"v1","addr":"172.21.15.16:6817","nonce":27851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1027851},{"type":"v1","addr":"172.21.15.16:6804","nonce":1027851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":27851},{"type":"v1","addr":"172.21.15.16:6823","nonce":27851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":27851},{"type":"v1","addr":"172.21.15.16:6821","nonce":27851}]},"public_addr":"172.21.15.16:6817/27851","cluster_addr":"172.21.15.16:6804/1027851","heartbeat_back_addr":"172.21.15.16:6823/27851","heartbeat_front_addr":"172.21.15.16:6821/27851","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2021-03-30T16:28:04.095151+0000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-03-30T15:44:42.849100+0000","dead_epoch":0},{"osd":1,"down_stamp":"2021-03-30T18:02:50.582257+0000","laggy_probability":0.50999999046325684,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-30T15:44:42.842032+0000","dead_epoch":38},{"osd":2,"down_stamp":"2021-03-30T16:49:03.087102+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-30T15:44:42.878038+0000","dead_epoch":33}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1560574643":"2021-03-31T16:32:33.417780+0000","172.21.15.16:6825/27507":"2021-03-31T16:32:33.417780+0000","172.21.15.16:6824/27507":"2021-03-31T16:32:33.417780+0000","172.21.15.16:0/3964122141":"2021-03-31T16:32:33.417780+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-30T18:49:18.947+0000 7f465a752700 1 -- 172.21.15.16:0/638134769 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f463402f2a0 msgr2=0x7f4634031760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-03-30T18:49:39.449+0000 7f187cc69700 1 -- 172.21.15.16:0/2192715209 learned_addr learned my addr 172.21.15.16:0/2192715209 (peer_addr_for_me v2:172.21.15.16:0/0) , 2021-03-30T18:49:39.450+0000 7f187cc69700 1 -- 172.21.15.16:0/2192715209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878113b20 con 0x7f18780894d0 , 2021-03-30T18:49:39.450+0000 7f187cc69700 1 --2- 172.21.15.16:0/2192715209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18780894d0 0x7f1878087a30 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f186c00c610 tx=0x7f186c007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:50:00.120+0000 7f7bf15c9700 1 --2- 172.21.15.16:0/4175043354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bec11f880 0x7f7bec121c70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f7bdc00b940 tx=0x7f7bdc006e20).ready entity=mon.0 client_cookie=bf414643afd5ed07 server_cookie=0 in_seq=0 out_seq=0 , 2021-03-30T18:50:00.120+0000 7f7bf0dc8700 1 -- 172.21.15.16:0/4175043354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bdc0097e0 con 0x7f7bec11f880 , 2021-03-30T18:50:00.120+0000 7f7bf0dc8700 1 -- 172.21.15.16:0/4175043354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bdc009940 con 0x7f7bec11f880 Ignoring malparsed backtrace: 2021-03-30T18:50:20.619+0000 7fc1528bd700 1 -- 172.21.15.16:0/3900037560 shutdown_connections , 2021-03-30T18:50:20.619+0000 7fc1528bd700 1 -- 172.21.15.16:0/3900037560 wait complete. , dumped all Ignoring malparsed backtrace: 2021-03-30T18:50:41.106+0000 7fee8b7fe700 1 -- 172.21.15.16:0/1812984149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fee94043bb0 con 0x7feea40663a0 , 2021-03-30T18:50:41.106+0000 7feea9c82700 1 -- 172.21.15.16:0/1812984149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea40683f0 con 0x7feea40663a0 , 2021-03-30T18:50:41.106+0000 7feea2ffd700 1 --2- 172.21.15.16:0/1812984149 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fee8402f2a0 0x7fee84031760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-03-30T18:51:01.765+0000 7fe12934c700 1 -- 172.21.15.16:0/3100663457 wait complete. , 2021-03-30T18:51:01.766+0000 7fe12934c700 1 Processor -- start , 2021-03-30T18:51:01.766+0000 7fe12934c700 1 -- start start Ignoring malparsed backtrace: 2021-03-30T18:51:22.431+0000 7f647effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64800888f0 0x7f6480088cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:51:22.431+0000 7f647effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64800888f0 0x7f6480088cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51116/0 (socket says 172.21.15.16:51116) , 2021-03-30T18:51:22.431+0000 7f647e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64800891c0 0x7f64800674b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44490/0 (socket says 172.21.15.16:44490) Ignoring malparsed backtrace: 2021-03-30T18:51:22.943+0000 7f200bfff700 1 -- 172.21.15.16:0/994882836 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f1fec02f2b0 msgr2=0x7f1fec031770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T18:51:22.943+0000 7f200bfff700 1 --2- 172.21.15.16:0/994882836 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f1fec02f2b0 0x7f1fec031770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1ffc0055d0 tx=0x7f1ffc007800).stop , 2021-03-30T18:51:22.943+0000 7f200bfff700 1 -- 172.21.15.16:0/994882836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c11b890 msgr2=0x7f200c082660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-03-30T18:51:43.445+0000 7fb0ea59c700 1 -- 172.21.15.16:0/54540666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec116460 con 0x7fb0ec0663a0 , 2021-03-30T18:51:43.445+0000 7fb0ea59c700 1 --2- 172.21.15.16:0/54540666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec0663a0 0x7fb0ec0899e0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fb0ec067eb0 tx=0x7fb0dc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2021-03-30T18:51:43.445+0000 7fb0da7fc700 1 -- 172.21.15.16:0/54540666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0dc00bee0 con 0x7fb0ec0663a0 Ignoring malparsed backtrace: 2021-03-30T18:52:04.108+0000 7f145b7fe700 1 --2- 172.21.15.16:0/213798947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c0888f0 0x7f145c088cd0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f144c00b940 tx=0x7f144c006e20).ready entity=mon.0 client_cookie=1b1b52955128bd2c server_cookie=0 in_seq=0 out_seq=0 , 2021-03-30T18:52:04.108+0000 7f145a7fc700 1 -- 172.21.15.16:0/213798947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f144c0097e0 con 0x7f145c0888f0 , 2021-03-30T18:52:04.108+0000 7f145a7fc700 1 -- 172.21.15.16:0/213798947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c009940 con 0x7f145c0888f0 Ignoring malparsed backtrace: 2021-03-30T18:52:24.617+0000 7fb3bf59e700 1 -- 172.21.15.16:0/1536675009 >> 172.21.15.16:0/1536675009 conn(0x7fb3c0062cf0 msgr2=0x7fb3c00650e0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:52:24.617+0000 7fb3bf59e700 1 -- 172.21.15.16:0/1536675009 shutdown_connections , 2021-03-30T18:52:24.617+0000 7fb3bf59e700 1 -- 172.21.15.16:0/1536675009 wait complete. Ignoring malparsed backtrace: 2021-03-30T18:52:45.103+0000 7fd975ffb700 1 -- 172.21.15.16:0/2477150591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7fd97801a060 con 0x7fd98811b150 , 2021-03-30T18:52:45.103+0000 7fd975ffb700 1 --2- 172.21.15.16:0/2477150591 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fd96802f2b0 0x7fd968031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:52:45.103+0000 7fd98fd19700 1 -- 172.21.15.16:0/2477150591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9880589e0 con 0x7fd98811b150 Ignoring malparsed backtrace: 2021-03-30T18:53:05.770+0000 7f72e5adf700 1 -- 172.21.15.16:0/2465097680 >> 172.21.15.16:0/2465097680 conn(0x7f72e00926f0 msgr2=0x7f72e008e540 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:53:05.770+0000 7f72e5adf700 1 -- 172.21.15.16:0/2465097680 shutdown_connections , 2021-03-30T18:53:05.770+0000 7f72e5adf700 1 -- 172.21.15.16:0/2465097680 wait complete. Ignoring malparsed backtrace: 2021-03-30T18:53:26.440+0000 7fc2a1bb2700 1 -- start start , 2021-03-30T18:53:26.440+0000 7fc2a1bb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc29c11dfb0 0x7fc29c1203a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:53:26.440+0000 7fc2a1bb2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c08d1c0 con 0x7fc29c11b150 Ignoring malparsed backtrace: 2021-03-30T18:53:26.958+0000 7f9f51916700 1 -- 172.21.15.16:0/3465498252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f4c120e40 con 0x7f9f4c07e230 , 2021-03-30T18:53:26.959+0000 7f9f42ffd700 1 -- 172.21.15.16:0/3465498252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+19110 (secure 0 0 0) 0x7f9f3c026ac0 con 0x7f9f4c07e230 , 2021-03-30T18:53:26.964 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2021-03-30T18:53:47.468+0000 7fab85068700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab800894d0 0x7fab80087a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:53:47.468+0000 7fab85068700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab800894d0 0x7fab80087a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51264/0 (socket says 172.21.15.16:51264) , 2021-03-30T18:53:47.468+0000 7fab85068700 1 -- 172.21.15.16:0/497017172 learned_addr learned my addr 172.21.15.16:0/497017172 (peer_addr_for_me v2:172.21.15.16:0/0) Ignoring malparsed backtrace: 2021-03-30T18:54:08.142+0000 7f7e2affd700 1 -- 172.21.15.16:0/2107706391 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0893e0 con 0x7f7e2c07e230 , 2021-03-30T18:54:08.142+0000 7f7e2a7fc700 1 -- 172.21.15.16:0/2107706391 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619289509 0 0) 0x7f7e2c0660d0 con 0x7f7e2c0888f0 , 2021-03-30T18:54:08.142+0000 7f7e2affd700 1 --2- 172.21.15.16:0/2107706391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c07e230 0x7f7e2c120880 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f7e2000b940 tx=0x7f7e20006e20).ready entity=mon.0 client_cookie=47284346ac11d577 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:54:28.638+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/3833633603 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f7a2c02f2b0 0x7f7a2c031770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:54:28.638+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/3833633603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c11b890 0x7f7a4c089b20 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:54:28.638+0000 7f7a4bfff700 1 -- 172.21.15.16:0/3833633603 >> 172.21.15.16:0/3833633603 conn(0x7f7a4c062cf0 msgr2=0x7f7a4c064bb0 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:54:49.126+0000 7f81ba7fc700 1 -- 172.21.15.16:0/2559521175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81bc01d700 con 0x7f81cc067c00 , 2021-03-30T18:54:49.126+0000 7f81cb59e700 1 -- 172.21.15.16:0/2559521175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc12b590 con 0x7f81cc067c00 , 2021-03-30T18:54:49.127+0000 7f81ba7fc700 1 -- 172.21.15.16:0/2559521175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7f81bc024030 con 0x7f81cc067c00 Ignoring malparsed backtrace: 2021-03-30T18:55:09.796+0000 7f328aefb700 1 -- 172.21.15.16:0/1974894226 shutdown_connections , 2021-03-30T18:55:09.796+0000 7f328aefb700 1 --2- 172.21.15.16:0/1974894226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328411b150 0x7f328411b530 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:55:09.796+0000 7f328aefb700 1 -- 172.21.15.16:0/1974894226 >> 172.21.15.16:0/1974894226 conn(0x7f328408c030 msgr2=0x7f328408c440 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:55:30.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-03-30T18:55:30.501+0000 7f415fcd2700 1 Processor -- start , 2021-03-30T18:55:30.502+0000 7f415fcd2700 1 -- start start Ignoring malparsed backtrace: 2021-03-30T18:55:30.864+0000 7f2deeffd700 1 -- 172.21.15.16:0/2599414153 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150898 (secure 0 0 0) 0x7f2de801b940 con 0x7f2df811b150 , 2021-03-30T18:55:31.021+0000 7f2dfd3ee700 1 -- 172.21.15.16:0/2599414153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2df811a6c0 con 0x7f2df811b150 , 2021-03-30T18:55:31.022+0000 7f2deeffd700 1 -- 172.21.15.16:0/2599414153 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+19110 (secure 0 0 0) 0x7f2de801b350 con 0x7f2df811b150 Ignoring malparsed backtrace: 2021-03-30T18:55:51.527+0000 7f2b63249700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c0894d0 0x7f2b5c0877c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:55:51.527+0000 7f2b63249700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c1136f0 con 0x7f2b5c0894d0 , 2021-03-30T18:55:51.527+0000 7f2b60fe5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c0894d0 0x7f2b5c0877c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-03-30T18:56:12.210+0000 7fd23bfff700 1 -- 172.21.15.16:0/2878007589 learned_addr learned my addr 172.21.15.16:0/2878007589 (peer_addr_for_me v1:172.21.15.16:0/0) , 2021-03-30T18:56:12.210+0000 7fd2408f6700 1 -- 172.21.15.16:0/2878007589 >> v1:172.21.15.16:6789/0 conn(0x7fd23c0663a0 legacy=0x7fd23c066830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T18:56:12.210+0000 7fd2408f6700 1 -- 172.21.15.16:0/2878007589 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c11acb0 con 0x7fd23c11b150 Ignoring malparsed backtrace: 2021-03-30T18:56:32.714+0000 7fd9c6f70700 1 -- 172.21.15.16:0/178485083 shutdown_connections , 2021-03-30T18:56:32.714+0000 7fd9c6f70700 1 --2- 172.21.15.16:0/178485083 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fd9a002f2b0 0x7fd9a0031770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:56:32.714+0000 7fd9c6f70700 1 --2- 172.21.15.16:0/178485083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c007f3a0 0x7fd9c0117fa0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-03-30T18:56:53.209+0000 7f52f0ff9700 1 -- 172.21.15.16:0/646304781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52e801d8e0 con 0x7f52f4065b10 , 2021-03-30T18:56:53.209+0000 7f52fafc9700 1 -- 172.21.15.16:0/646304781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f412b390 con 0x7f52f4065b10 , 2021-03-30T18:56:53.209+0000 7f52fafc9700 1 -- 172.21.15.16:0/646304781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f412b7a0 con 0x7f52f4065b10 Ignoring malparsed backtrace: 2021-03-30T18:57:13.870+0000 7fcde9ff3700 1 --2- 172.21.15.16:0/2637255144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde411b890 0x7fcde411bcb0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:57:13.870+0000 7fcde9ff3700 1 -- 172.21.15.16:0/2637255144 >> 172.21.15.16:0/2637255144 conn(0x7fcde4062cf0 msgr2=0x7fcde4065110 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:57:13.870+0000 7fcde9ff3700 1 -- 172.21.15.16:0/2637255144 shutdown_connections Ignoring malparsed backtrace: 2021-03-30T18:57:34.546+0000 7fc66eac2700 1 -- start start , 2021-03-30T18:57:34.546+0000 7fc66eac2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6680676b0 0x7fc66811fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:57:34.546+0000 7fc66eac2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66807e810 con 0x7fc668067c00 Ignoring malparsed backtrace: 2021-03-30T18:57:35.057+0000 7fec11ffe700 1 -- 172.21.15.16:0/2190954222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec0c081fc0 con 0x7fec0c0663a0 , 2021-03-30T18:57:35.058+0000 7febf37fe700 1 -- 172.21.15.16:0/2190954222 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+19110 (secure 0 0 0) 0x7febfc00fe60 con 0x7fec0c0663a0 , 2021-03-30T18:57:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2021-03-30T18:57:55.574+0000 7f4febcc0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4117790 con 0x7f4fe411b150 , 2021-03-30T18:57:55.574+0000 7f4fe9a5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fe411b150 0x7f4fe4089780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:57:55.574+0000 7f4fe9a5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fe411b150 0x7f4fe4089780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51504/0 (socket says 172.21.15.16:51504) Ignoring malparsed backtrace: 2021-03-30T18:58:16.237+0000 7f174dd83700 1 -- 172.21.15.16:0/1766276438 >> v1:172.21.15.16:6789/0 conn(0x7f17480663a0 legacy=0x7f1748066830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T18:58:16.237+0000 7f174dd83700 1 -- 172.21.15.16:0/1766276438 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174811acb0 con 0x7f174811b150 , 2021-03-30T18:58:16.237+0000 7f174cd81700 1 -- 172.21.15.16:0/1766276438 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301879355 0 0) 0x7f17480680e0 con 0x7f17480663a0 Ignoring malparsed backtrace: 2021-03-30T18:58:36.747+0000 7fc7ea873700 1 --2- 172.21.15.16:0/3563008885 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fc7c402f2a0 0x7fc7c4031760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:58:36.747+0000 7fc7ea873700 1 --2- 172.21.15.16:0/3563008885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e4065d70 0x7fc7e412d300 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:58:36.747+0000 7fc7ea873700 1 -- 172.21.15.16:0/3563008885 >> 172.21.15.16:0/3563008885 conn(0x7fc7e40926f0 msgr2=0x7fc7e40654b0 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:58:57.262+0000 7f31eedc5700 1 -- 172.21.15.16:0/1355447012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e8066e90 con 0x7f31e80888f0 , 2021-03-30T18:58:57.263+0000 7f31e4ff9700 1 -- 172.21.15.16:0/1355447012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7f31dc00f070 con 0x7f31e80888f0 , 2021-03-30T18:58:57.263+0000 7f31e4ff9700 1 --2- 172.21.15.16:0/1355447012 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f31c802f2b0 0x7f31c8031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-03-30T18:59:18.036+0000 7f8cfd667700 1 --2- 172.21.15.16:0/1541290232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf811dfb0 0x7f8cf81203a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7f8ce800b940 tx=0x7f8ce8006e20).stop , 2021-03-30T18:59:18.037+0000 7f8cfd667700 1 -- 172.21.15.16:0/1541290232 shutdown_connections , 2021-03-30T18:59:18.037+0000 7f8cfd667700 1 --2- 172.21.15.16:0/1541290232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf811dfb0 0x7f8cf81203a0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-03-30T18:59:18.667 INFO:tasks.ceph:Waiting for all PGs to be active+clean and split+merged, waiting on ['3.3f', '4.38', '3.3e', '4.39', '3.3d', '4.3a', '3.3c', '4.3b', '3.3b', '4.3c', '3.3a', '4.3d', '3.39', '4.3e', '3.38', '4.3f', '3.37', '4.30', '3.36', '4.31', '3.35', '4.32', '3.34', '4.33', '3.33', '4.34', '3.32', '4.35', '3.31', '4.36', '3.30', '4.37', '3.2f', '4.28', '3.2e', '4.29', '3.2d', '4.2a', '3.2c', '4.2b', '3.2b', '4.2c', '3.2a', '4.2d', '3.29', '4.2e', '3.28', '4.2f', '3.27', '4.20', '3.26', '4.21', '3.25', '4.22', '3.24', '4.23', '3.23', '4.24', '3.22', '4.25', '3.21', '4.26', '3.20', '4.27', '9.15', '8.14', '7.1b', '6.1a', '3.1f', '4.18', '5.19', '9.14', '8.15', '7.1a', '6.1b', '3.1e', '4.19', '5.18', '9.17', '8.16', '7.19', '6.18', '3.1d', '4.1a', '5.1b', '9.16', '8.17', '7.18', '6.19', '3.1c', '4.1b', '5.1a', '9.11', '8.10', '7.1f', '6.1e', '5.1d', '3.1b', '4.1c', '9.c', '8.d', '7.2', '6.3', '4.1', '3.6', '2.7', '5.0', '9.d', '8.c', '7.3', '6.2', '4.0', '2.6', '3.7', '5.1', '9.e', '8.f', '7.0', '6.1', '4.3', '3.4', '2.5', '5.2', '9.b', '8.a', '7.5', '6.4', '3.1', '2.0', '4.6', '5.7', '9.8', '8.9', '7.6', '6.7', '3.2', '1.0', '2.3', '4.5', '5.4', '9.a', '8.b', '7.4', '6.5', '3.0', '2.1', '4.7', '5.6', '9.9', '8.8', '7.7', '6.6', '4.4', '3.3', '2.2', '5.5', '9.f', '8.e', '7.1', '6.0', '4.2', '3.5', '2.4', '5.3', '9.2', '8.3', '7.c', '6.d', '3.8', '4.f', '5.e', '9.3', '8.2', '7.d', '6.c', '3.9', '4.e', '5.f', '9.0', '8.1', '7.e', '6.f', '3.a', '4.d', '5.c', '9.1', '8.0', '7.f', '6.e', '3.b', '4.c', '5.d', '9.6', '8.7', '7.8', '6.9', '3.c', '4.b', '5.a', '9.7', '8.6', '7.9', '6.8', '3.d', '4.a', '5.b', '9.4', '8.5', '7.a', '6.b', '3.e', '4.9', '5.8', '9.5', '8.4', '7.b', '6.a', '3.f', '4.8', '5.9', '9.1a', '8.1b', '7.14', '6.15', '3.10', '4.17', '5.16', '9.1b', '8.1a', '7.15', '6.14', '3.11', '4.16', '5.17', '9.18', '8.19', '7.16', '6.17', '3.12', '4.15', '5.14', '9.19', '8.18', '7.17', '6.16', '3.13', '4.14', '5.15', '9.1e', '8.1f', '7.10', '6.11', '3.14', '4.13', '5.12', '9.1f', '8.1e', '7.11', '6.10', '3.15', '4.12', '5.13', '9.1c', '8.1d', '7.12', '6.13', '3.16', '4.11', '5.10', '9.1d', '8.1c', '7.13', '6.12', '3.17', '4.10', '5.11', '9.12', '8.13', '7.1c', '6.1d', '5.1e', '3.18', '4.1f', '9.13', '8.12', '7.1d', '6.1c', '5.1f', '3.19', '4.1e', '9.10', '8.11', '7.1e', '6.1f', '3.1a', '4.1d', '5.1c'] to go clean , 2021-03-30T18:59:38.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-03-30T18:59:38.808+0000 7f82b7ae1700 1 Processor -- start Ignoring malparsed backtrace: 2021-03-30T18:59:39.213+0000 7ff533fff700 1 -- 172.21.15.16:0/3617875649 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5858+0+0 (secure 0 0 0) 0x7ff53c03f9b0 con 0x7ff54c066650 , 2021-03-30T18:59:39.213+0000 7ff54b7fe700 1 --2- 172.21.15.16:0/3617875649 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7ff52c02f2b0 0x7ff52c031770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:59:39.213+0000 7ff54b7fe700 1 --2- 172.21.15.16:0/3617875649 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7ff52c02f2b0 0x7ff52c031770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff54000b940 tx=0x7ff540016000).ready entity=mgr.4248 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:59:59.916+0000 7f8b2384d700 1 -- start start , 2021-03-30T18:59:59.916+0000 7f8b2384d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b1c065e90 0x7f8b1c0882a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:59:59.916+0000 7f8b2384d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c087330 con 0x7f8b1c065e90 Ignoring malparsed backtrace: 2021-03-30T19:00:20.589+0000 7fcd1ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd2011c1a0 0x7fcd2007e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45014/0 (socket says 172.21.15.16:45014) , 2021-03-30T19:00:20.589+0000 7fcd1ffff700 1 -- 172.21.15.16:0/1833495911 learned_addr learned my addr 172.21.15.16:0/1833495911 (peer_addr_for_me v1:172.21.15.16:0/0) , 2021-03-30T19:00:20.590+0000 7fcd248dc700 1 -- 172.21.15.16:0/1833495911 >> v1:172.21.15.16:6789/0 conn(0x7fcd2011c1a0 legacy=0x7fcd2007e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-03-30T19:00:41.100+0000 7ff943a61700 1 -- 172.21.15.16:0/3921928641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff93c064930 msgr2=0x7ff93c120df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T19:00:41.100+0000 7ff943a61700 1 --2- 172.21.15.16:0/3921928641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff93c064930 0x7ff93c120df0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7ff93800b150 tx=0x7ff938016000).stop , 2021-03-30T19:00:41.100+0000 7ff943a61700 1 -- 172.21.15.16:0/3921928641 shutdown_connections Ignoring malparsed backtrace: 2021-03-30T19:01:01.592+0000 7fb1937fe700 1 -- 172.21.15.16:0/2384871957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb19c0054f0 con 0x7fb1ac0676b0 , 2021-03-30T19:01:01.592+0000 7fb1937fe700 1 -- 172.21.15.16:0/2384871957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c005650 con 0x7fb1ac0676b0 , 2021-03-30T19:01:01.592+0000 7fb1b0896700 1 -- 172.21.15.16:0/2384871957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac080630 con 0x7fb1ac0676b0 Ignoring malparsed backtrace: 2021-03-30T19:01:22.266+0000 7f687d59a700 1 -- 172.21.15.16:0/746905030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6870009bf0 con 0x7f688011b150 , 2021-03-30T19:01:22.266+0000 7f6884967700 1 -- 172.21.15.16:0/746905030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f688011b150 msgr2=0x7f688011b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T19:01:22.266+0000 7f6884967700 1 --2- 172.21.15.16:0/746905030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f688011b150 0x7f688011b530 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f687000b940 tx=0x7f6870006e20).stop Ignoring malparsed backtrace: 2021-03-30T19:01:22.790+0000 7f570b252700 1 -- 172.21.15.16:0/1125698364 wait complete. , 2021-03-30T19:01:22.805 INFO:tasks.ceph:Waiting for all PGs to be active+clean and split+merged, waiting on ['3.3f', '4.38', '3.3e', '4.39', '3.3d', '4.3a', '3.3c', '4.3b', '3.3b', '4.3c', '3.3a', '4.3d', '3.39', '4.3e', '3.38', '4.3f', '3.37', '4.30', '3.36', '4.31', '3.35', '4.32', '3.34', '4.33', '3.33', '4.34', '3.32', '4.35', '3.31', '4.36', '3.30', '4.37', '3.2f', '4.28', '3.2e', '4.29', '3.2d', '4.2a', '3.2c', '4.2b', '3.2b', '4.2c', '3.2a', '4.2d', '3.29', '4.2e', '3.28', '4.2f', '3.27', '4.20', '3.26', '4.21', '3.25', '4.22', '3.24', '4.23', '3.23', '4.24', '3.22', '4.25', '3.21', '4.26', '3.20', '4.27', '9.15', '8.14', '7.1b', '6.1a', '3.1f', '4.18', '5.19', '9.14', '8.15', '7.1a', '6.1b', '3.1e', '4.19', '5.18', '9.17', '8.16', '7.19', '6.18', '3.1d', '4.1a', '5.1b', '9.16', '8.17', '7.18', '6.19', '3.1c', '4.1b', '5.1a', '9.11', '8.10', '7.1f', '6.1e', '5.1d', '3.1b', '4.1c', '9.c', '8.d', '7.2', '6.3', '4.1', '3.6', '2.7', '5.0', '9.d', '8.c', '7.3', '6.2', '4.0', '2.6', '3.7', '5.1', '9.e', '8.f', '7.0', '6.1', '4.3', '3.4', '2.5', '5.2', '9.b', '8.a', '7.5', '6.4', '3.1', '2.0', '4.6', '5.7', '9.8', '8.9', '7.6', '6.7', '3.2', '1.0', '2.3', '4.5', '5.4', '9.a', '8.b', '7.4', '6.5', '3.0', '2.1', '4.7', '5.6', '9.9', '8.8', '7.7', '6.6', '4.4', '3.3', '2.2', '5.5', '9.f', '8.e', '7.1', '6.0', '4.2', '3.5', '2.4', '5.3', '9.2', '8.3', '7.c', '6.d', '3.8', '4.f', '5.e', '9.3', '8.2', '7.d', '6.c', '3.9', '4.e', '5.f', '9.0', '8.1', '7.e', '6.f', '3.a', '4.d', '5.c', '9.1', '8.0', '7.f', '6.e', '3.b', '4.c', '5.d', '9.6', '8.7', '7.8', '6.9', '3.c', '4.b', '5.a', '9.7', '8.6', '7.9', '6.8', '3.d', '4.a', '5.b', '9.4', '8.5', '7.a', '6.b', '3.e', '4.9', '5.8', '9.5', '8.4', '7.b', '6.a', '3.f', '4.8', '5.9', '9.1a', '8.1b', '7.14', '6.15', '3.10', '4.17', '5.16', '9.1b', '8.1a', '7.15', '6.14', '3.11', '4.16', '5.17', '9.18', '8.19', '7.16', '6.17', '3.12', '4.15', '5.14', '9.19', '8.18', '7.17', '6.16', '3.13', '4.14', '5.15', '9.1e', '8.1f', '7.10', '6.11', '3.14', '4.13', '5.12', '9.1f', '8.1e', '7.11', '6.10', '3.15', '4.12', '5.13', '9.1c', '8.1d', '7.12', '6.13', '3.16', '4.11', '5.10', '9.1d', '8.1c', '7.13', '6.12', '3.17', '4.10', '5.11', '9.12', '8.13', '7.1c', '6.1d', '5.1e', '3.18', '4.1f', '9.13', '8.12', '7.1d', '6.1c', '5.1f', '3.19', '4.1e', '9.10', '8.11', '7.1e', '6.1f', '3.1a', '4.1d', '5.1c'] to go clean , 2021-03-30T19:01:42.807 ERROR:teuthology.contextutil:Saw exception from nested tasks Ignoring malparsed backtrace: 2021-03-30T19:01:42.924 INFO:teuthology.misc:Shutting down mon daemons... , 2021-03-30T19:01:42.925 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-03-30T19:01:42.925 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/task/internal/__init__.py", line 463, in coredump , yield Not-understood crash indication Binary file /home/teuthworker/archive/dang-2021-03-30_07:32:10-rgw-wip-dang-zipper-user-distro-basic-smithi/6011473/teuthology.log matches Not-understood crash indication Ignoring malparsed backtrace: ceph version 17.0.0-2245-ge841aa06 (e841aa06e2fa775b691b12cf67756986cdba1952) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x558512e59504] , 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x4c9) [0x5585135bd549] Ignoring malparsed backtrace: 2021-03-30T16:51:22.765+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:51:23.754+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:51:24.748+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:53:33.635+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:53:34.605+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:53:35.577+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:55:14.667+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:55:15.672+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:55:16.631+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:56:55.444+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:56:56.490+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:56:57.498+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 28 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T16:58:36.498+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T16:58:37.547+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) , 2021-03-30T16:58:38.553+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:00:17.642+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:00:18.671+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:00:19.665+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:01:58.818+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:01:59.830+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:02:00.852+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:03:39.886+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:03:40.919+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:03:41.930+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:05:20.732+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:05:21.698+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:05:22.650+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 51 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:07:01.683+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:07:02.681+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:07:03.687+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:08:43.185+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:08:44.217+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:08:45.235+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:10:24.535+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:10:25.500+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:10:26.476+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:12:04.972+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:12:06.010+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:12:06.978+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 42 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:13:46.152+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:13:47.120+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:13:48.127+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:15:27.577+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:15:28.613+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:15:29.569+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:17:08.494+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:17:09.457+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:17:10.418+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:18:49.554+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:18:50.595+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:18:51.554+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:20:31.033+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:20:32.027+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:20:33.046+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:22:11.592+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:22:12.600+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:22:13.616+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 76 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:23:52.798+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:23:53.829+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:23:54.853+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:25:34.283+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:25:35.321+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:25:36.292+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:27:17.161+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:27:18.144+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:27:19.180+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:29:42.863+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 94 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:29:44.336+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 94 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:29:45.366+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 94 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:31:25.504+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:31:26.503+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:31:27.536+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4243.0:634650 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:33:08.802+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:33:09.835+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:33:11.111+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4243.0:634597 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T17:39:20.833+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:39:37.239+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4243.0:634946 7.3 7.31099063 (undecoded) ondisk+write+known_if_redirected e40) , 2021-03-30T17:39:55.083+0000 7fa656e8f700 -1 osd.1 40 get_health_metrics reporting 157 slow ops, oldest is osd_op(client.4245.0:5737598 3.26 3.3a77dea6 (undecoded) ondisk+read+known_if_redirected e40) Ignoring malparsed backtrace: 2021-03-30T18:47:55.413+0000 7f530d175700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530009b370 con 0x7f53000ac050 , 2021-03-30T18:47:55.414+0000 7f5307fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53000ac050 0x7f53000b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:47:55.414+0000 7f5307fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53000ac050 0x7f53000b2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50908/0 (socket says 172.21.15.16:50908) Ignoring malparsed backtrace: 2021-03-30T18:47:56.093+0000 7fa1f67fc700 1 -- 172.21.15.16:0/3471896784 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522657018 0 0) 0x7fa1f808d1c0 con 0x7fa1f811b150 , 2021-03-30T18:47:56.093+0000 7fa1f6ffd700 1 -- 172.21.15.16:0/3471896784 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f811acb0 con 0x7fa1f811dfb0 , 2021-03-30T18:47:56.093+0000 7fa1f6ffd700 1 --2- 172.21.15.16:0/3471896784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f811dfb0 0x7fa1f81203a0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7fa1ec00b940 tx=0x7fa1ec006e20).ready entity=mon.0 client_cookie=89e6bcd86f57b914 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:48:16.599+0000 7f99d1027700 1 --2- 172.21.15.16:0/2825235693 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f99b002f2b0 0x7f99b0031770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:48:16.599+0000 7f99d1027700 1 --2- 172.21.15.16:0/2825235693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99cc088630 0x7f99cc12d150 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:48:16.599+0000 7f99d1027700 1 -- 172.21.15.16:0/2825235693 >> 172.21.15.16:0/2825235693 conn(0x7f99cc08b660 msgr2=0x7f99cc066a00 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:48:37.093+0000 7f81f6ffd700 1 -- 172.21.15.16:0/1493163775 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7f81f000f070 con 0x7f8200089850 , 2021-03-30T18:48:37.093+0000 7f81fffff700 1 -- 172.21.15.16:0/1493163775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82001188e0 con 0x7f8200089850 , 2021-03-30T18:48:37.094+0000 7f81f6ffd700 1 --2- 172.21.15.16:0/1493163775 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f81e402f2b0 0x7f81e4031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-03-30T18:48:57.763+0000 7fa6c0943700 1 -- 172.21.15.16:0/286481548 >> 172.21.15.16:0/286481548 conn(0x7fa6bc062cf0 msgr2=0x7fa6bc065110 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:48:57.763+0000 7fa6c0943700 1 -- 172.21.15.16:0/286481548 shutdown_connections , 2021-03-30T18:48:57.764+0000 7fa6c0943700 1 -- 172.21.15.16:0/286481548 wait complete. Ignoring malparsed backtrace: 2021-03-30T18:49:18.425+0000 7faf20e46700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1c08f380 con 0x7faf1c066650 , 2021-03-30T18:49:18.425+0000 7faf20e46700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1c068660 con 0x7faf1c121b20 , 2021-03-30T18:49:18.425+0000 7faf1affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf1c121b20 0x7faf1c123f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-03-30T18:49:18.948 INFO:teuthology.orchestra.run.smithi016.stdout: , 2021-03-30T18:49:18.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"0b4129c0-5322-4cef-8ab1-aa1d5b1c34aa","created":"2021-03-30T15:44:37.569199+0000","modified":"2021-03-30T18:02:50.582257+0000","last_up_change":"2021-03-30T18:02:50.582257+0000","last_in_change":"2021-03-30T16:41:52.416862+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-30T15:44:43.950286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-30T15:44:46.057469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-03-30T15:44:55.025694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-03-30T15:44:57.303437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-03-30T15:44:59.165384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-03-30T15:45:01.036872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-03-30T15:45:03.014361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-03-30T15:45:05.275172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.buckets.non-ec","create_time":"2021-03-30T15:46:14.040116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4f3b9446-e5a3-4faf-b50f-0fbc3c82709a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":33,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":27827},{"type":"v1","addr":"172.21.15.16:6805","nonce":27827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":27827},{"type":"v1","addr":"172.21.15.16:6809","nonce":27827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":27827},{"type":"v1","addr":"172.21.15.16:6815","nonce":27827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":27827},{"type":"v1","addr":"172.21.15.16:6813","nonce":27827}]},"public_addr":"172.21.15.16:6805/27827","cluster_addr":"172.21.15.16:6809/27827","heartbeat_back_addr":"172.21.15.16:6815/27827","heartbeat_front_addr":"172.21.15.16:6813/27827","state":["autoout","exists"]},{"osd":1,"uuid":"3f7e8b67-ed65-4e68-841a-9cc5b6176b97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":38,"up_from":39,"up_thru":39,"down_at":41,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":27828},{"type":"v1","addr":"172.21.15.16:6801","nonce":27828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2027828},{"type":"v1","addr":"172.21.15.16:6804","nonce":2027828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":27828},{"type":"v1","addr":"172.21.15.16:6812","nonce":27828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":27828},{"type":"v1","addr":"172.21.15.16:6808","nonce":27828}]},"public_addr":"172.21.15.16:6801/27828","cluster_addr":"172.21.15.16:6804/2027828","heartbeat_back_addr":"172.21.15.16:6812/27828","heartbeat_front_addr":"172.21.15.16:6808/27828","state":["exists"]},{"osd":2,"uuid":"6cfa9335-1f9b-4d97-b631-847ef2077610","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":33,"up_from":34,"up_thru":34,"down_at":38,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":27851},{"type":"v1","addr":"172.21.15.16:6817","nonce":27851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1027851},{"type":"v1","addr":"172.21.15.16:6804","nonce":1027851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":27851},{"type":"v1","addr":"172.21.15.16:6823","nonce":27851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":27851},{"type":"v1","addr":"172.21.15.16:6821","nonce":27851}]},"public_addr":"172.21.15.16:6817/27851","cluster_addr":"172.21.15.16:6804/1027851","heartbeat_back_addr":"172.21.15.16:6823/27851","heartbeat_front_addr":"172.21.15.16:6821/27851","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2021-03-30T16:28:04.095151+0000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-03-30T15:44:42.849100+0000","dead_epoch":0},{"osd":1,"down_stamp":"2021-03-30T18:02:50.582257+0000","laggy_probability":0.50999999046325684,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-30T15:44:42.842032+0000","dead_epoch":38},{"osd":2,"down_stamp":"2021-03-30T16:49:03.087102+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-30T15:44:42.878038+0000","dead_epoch":33}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1560574643":"2021-03-31T16:32:33.417780+0000","172.21.15.16:6825/27507":"2021-03-31T16:32:33.417780+0000","172.21.15.16:6824/27507":"2021-03-31T16:32:33.417780+0000","172.21.15.16:0/3964122141":"2021-03-31T16:32:33.417780+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-30T18:49:18.947+0000 7f465a752700 1 -- 172.21.15.16:0/638134769 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f463402f2a0 msgr2=0x7f4634031760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-03-30T18:49:39.449+0000 7f187cc69700 1 -- 172.21.15.16:0/2192715209 learned_addr learned my addr 172.21.15.16:0/2192715209 (peer_addr_for_me v2:172.21.15.16:0/0) , 2021-03-30T18:49:39.450+0000 7f187cc69700 1 -- 172.21.15.16:0/2192715209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878113b20 con 0x7f18780894d0 , 2021-03-30T18:49:39.450+0000 7f187cc69700 1 --2- 172.21.15.16:0/2192715209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18780894d0 0x7f1878087a30 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f186c00c610 tx=0x7f186c007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:50:00.120+0000 7f7bf15c9700 1 --2- 172.21.15.16:0/4175043354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bec11f880 0x7f7bec121c70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f7bdc00b940 tx=0x7f7bdc006e20).ready entity=mon.0 client_cookie=bf414643afd5ed07 server_cookie=0 in_seq=0 out_seq=0 , 2021-03-30T18:50:00.120+0000 7f7bf0dc8700 1 -- 172.21.15.16:0/4175043354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bdc0097e0 con 0x7f7bec11f880 , 2021-03-30T18:50:00.120+0000 7f7bf0dc8700 1 -- 172.21.15.16:0/4175043354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bdc009940 con 0x7f7bec11f880 Ignoring malparsed backtrace: 2021-03-30T18:50:20.619+0000 7fc1528bd700 1 -- 172.21.15.16:0/3900037560 shutdown_connections , 2021-03-30T18:50:20.619+0000 7fc1528bd700 1 -- 172.21.15.16:0/3900037560 wait complete. , dumped all Ignoring malparsed backtrace: 2021-03-30T18:50:41.106+0000 7fee8b7fe700 1 -- 172.21.15.16:0/1812984149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fee94043bb0 con 0x7feea40663a0 , 2021-03-30T18:50:41.106+0000 7feea9c82700 1 -- 172.21.15.16:0/1812984149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea40683f0 con 0x7feea40663a0 , 2021-03-30T18:50:41.106+0000 7feea2ffd700 1 --2- 172.21.15.16:0/1812984149 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fee8402f2a0 0x7fee84031760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-03-30T18:51:01.765+0000 7fe12934c700 1 -- 172.21.15.16:0/3100663457 wait complete. , 2021-03-30T18:51:01.766+0000 7fe12934c700 1 Processor -- start , 2021-03-30T18:51:01.766+0000 7fe12934c700 1 -- start start Ignoring malparsed backtrace: 2021-03-30T18:51:22.431+0000 7f647effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64800888f0 0x7f6480088cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:51:22.431+0000 7f647effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64800888f0 0x7f6480088cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51116/0 (socket says 172.21.15.16:51116) , 2021-03-30T18:51:22.431+0000 7f647e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64800891c0 0x7f64800674b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44490/0 (socket says 172.21.15.16:44490) Ignoring malparsed backtrace: 2021-03-30T18:51:22.943+0000 7f200bfff700 1 -- 172.21.15.16:0/994882836 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f1fec02f2b0 msgr2=0x7f1fec031770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T18:51:22.943+0000 7f200bfff700 1 --2- 172.21.15.16:0/994882836 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f1fec02f2b0 0x7f1fec031770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1ffc0055d0 tx=0x7f1ffc007800).stop , 2021-03-30T18:51:22.943+0000 7f200bfff700 1 -- 172.21.15.16:0/994882836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f200c11b890 msgr2=0x7f200c082660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-03-30T18:51:43.445+0000 7fb0ea59c700 1 -- 172.21.15.16:0/54540666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec116460 con 0x7fb0ec0663a0 , 2021-03-30T18:51:43.445+0000 7fb0ea59c700 1 --2- 172.21.15.16:0/54540666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0ec0663a0 0x7fb0ec0899e0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fb0ec067eb0 tx=0x7fb0dc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 , 2021-03-30T18:51:43.445+0000 7fb0da7fc700 1 -- 172.21.15.16:0/54540666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0dc00bee0 con 0x7fb0ec0663a0 Ignoring malparsed backtrace: 2021-03-30T18:52:04.108+0000 7f145b7fe700 1 --2- 172.21.15.16:0/213798947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f145c0888f0 0x7f145c088cd0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f144c00b940 tx=0x7f144c006e20).ready entity=mon.0 client_cookie=1b1b52955128bd2c server_cookie=0 in_seq=0 out_seq=0 , 2021-03-30T18:52:04.108+0000 7f145a7fc700 1 -- 172.21.15.16:0/213798947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f144c0097e0 con 0x7f145c0888f0 , 2021-03-30T18:52:04.108+0000 7f145a7fc700 1 -- 172.21.15.16:0/213798947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c009940 con 0x7f145c0888f0 Ignoring malparsed backtrace: 2021-03-30T18:52:24.617+0000 7fb3bf59e700 1 -- 172.21.15.16:0/1536675009 >> 172.21.15.16:0/1536675009 conn(0x7fb3c0062cf0 msgr2=0x7fb3c00650e0 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:52:24.617+0000 7fb3bf59e700 1 -- 172.21.15.16:0/1536675009 shutdown_connections , 2021-03-30T18:52:24.617+0000 7fb3bf59e700 1 -- 172.21.15.16:0/1536675009 wait complete. Ignoring malparsed backtrace: 2021-03-30T18:52:45.103+0000 7fd975ffb700 1 -- 172.21.15.16:0/2477150591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7fd97801a060 con 0x7fd98811b150 , 2021-03-30T18:52:45.103+0000 7fd975ffb700 1 --2- 172.21.15.16:0/2477150591 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fd96802f2b0 0x7fd968031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:52:45.103+0000 7fd98fd19700 1 -- 172.21.15.16:0/2477150591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9880589e0 con 0x7fd98811b150 Ignoring malparsed backtrace: 2021-03-30T18:53:05.770+0000 7f72e5adf700 1 -- 172.21.15.16:0/2465097680 >> 172.21.15.16:0/2465097680 conn(0x7f72e00926f0 msgr2=0x7f72e008e540 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:53:05.770+0000 7f72e5adf700 1 -- 172.21.15.16:0/2465097680 shutdown_connections , 2021-03-30T18:53:05.770+0000 7f72e5adf700 1 -- 172.21.15.16:0/2465097680 wait complete. Ignoring malparsed backtrace: 2021-03-30T18:53:26.440+0000 7fc2a1bb2700 1 -- start start , 2021-03-30T18:53:26.440+0000 7fc2a1bb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc29c11dfb0 0x7fc29c1203a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:53:26.440+0000 7fc2a1bb2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c08d1c0 con 0x7fc29c11b150 Ignoring malparsed backtrace: 2021-03-30T18:53:26.958+0000 7f9f51916700 1 -- 172.21.15.16:0/3465498252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f4c120e40 con 0x7f9f4c07e230 , 2021-03-30T18:53:26.959+0000 7f9f42ffd700 1 -- 172.21.15.16:0/3465498252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+19110 (secure 0 0 0) 0x7f9f3c026ac0 con 0x7f9f4c07e230 , 2021-03-30T18:53:26.964 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2021-03-30T18:53:47.468+0000 7fab85068700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab800894d0 0x7fab80087a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:53:47.468+0000 7fab85068700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab800894d0 0x7fab80087a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51264/0 (socket says 172.21.15.16:51264) , 2021-03-30T18:53:47.468+0000 7fab85068700 1 -- 172.21.15.16:0/497017172 learned_addr learned my addr 172.21.15.16:0/497017172 (peer_addr_for_me v2:172.21.15.16:0/0) Ignoring malparsed backtrace: 2021-03-30T18:54:08.142+0000 7f7e2affd700 1 -- 172.21.15.16:0/2107706391 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0893e0 con 0x7f7e2c07e230 , 2021-03-30T18:54:08.142+0000 7f7e2a7fc700 1 -- 172.21.15.16:0/2107706391 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619289509 0 0) 0x7f7e2c0660d0 con 0x7f7e2c0888f0 , 2021-03-30T18:54:08.142+0000 7f7e2affd700 1 --2- 172.21.15.16:0/2107706391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c07e230 0x7f7e2c120880 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f7e2000b940 tx=0x7f7e20006e20).ready entity=mon.0 client_cookie=47284346ac11d577 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:54:28.638+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/3833633603 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f7a2c02f2b0 0x7f7a2c031770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:54:28.638+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/3833633603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c11b890 0x7f7a4c089b20 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:54:28.638+0000 7f7a4bfff700 1 -- 172.21.15.16:0/3833633603 >> 172.21.15.16:0/3833633603 conn(0x7f7a4c062cf0 msgr2=0x7f7a4c064bb0 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:54:49.126+0000 7f81ba7fc700 1 -- 172.21.15.16:0/2559521175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81bc01d700 con 0x7f81cc067c00 , 2021-03-30T18:54:49.126+0000 7f81cb59e700 1 -- 172.21.15.16:0/2559521175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc12b590 con 0x7f81cc067c00 , 2021-03-30T18:54:49.127+0000 7f81ba7fc700 1 -- 172.21.15.16:0/2559521175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7f81bc024030 con 0x7f81cc067c00 Ignoring malparsed backtrace: 2021-03-30T18:55:09.796+0000 7f328aefb700 1 -- 172.21.15.16:0/1974894226 shutdown_connections , 2021-03-30T18:55:09.796+0000 7f328aefb700 1 --2- 172.21.15.16:0/1974894226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f328411b150 0x7f328411b530 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:55:09.796+0000 7f328aefb700 1 -- 172.21.15.16:0/1974894226 >> 172.21.15.16:0/1974894226 conn(0x7f328408c030 msgr2=0x7f328408c440 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:55:30.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-03-30T18:55:30.501+0000 7f415fcd2700 1 Processor -- start , 2021-03-30T18:55:30.502+0000 7f415fcd2700 1 -- start start Ignoring malparsed backtrace: 2021-03-30T18:55:30.864+0000 7f2deeffd700 1 -- 172.21.15.16:0/2599414153 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+150898 (secure 0 0 0) 0x7f2de801b940 con 0x7f2df811b150 , 2021-03-30T18:55:31.021+0000 7f2dfd3ee700 1 -- 172.21.15.16:0/2599414153 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2df811a6c0 con 0x7f2df811b150 , 2021-03-30T18:55:31.022+0000 7f2deeffd700 1 -- 172.21.15.16:0/2599414153 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+19110 (secure 0 0 0) 0x7f2de801b350 con 0x7f2df811b150 Ignoring malparsed backtrace: 2021-03-30T18:55:51.527+0000 7f2b63249700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c0894d0 0x7f2b5c0877c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:55:51.527+0000 7f2b63249700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c1136f0 con 0x7f2b5c0894d0 , 2021-03-30T18:55:51.527+0000 7f2b60fe5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b5c0894d0 0x7f2b5c0877c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 Ignoring malparsed backtrace: 2021-03-30T18:56:12.210+0000 7fd23bfff700 1 -- 172.21.15.16:0/2878007589 learned_addr learned my addr 172.21.15.16:0/2878007589 (peer_addr_for_me v1:172.21.15.16:0/0) , 2021-03-30T18:56:12.210+0000 7fd2408f6700 1 -- 172.21.15.16:0/2878007589 >> v1:172.21.15.16:6789/0 conn(0x7fd23c0663a0 legacy=0x7fd23c066830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T18:56:12.210+0000 7fd2408f6700 1 -- 172.21.15.16:0/2878007589 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c11acb0 con 0x7fd23c11b150 Ignoring malparsed backtrace: 2021-03-30T18:56:32.714+0000 7fd9c6f70700 1 -- 172.21.15.16:0/178485083 shutdown_connections , 2021-03-30T18:56:32.714+0000 7fd9c6f70700 1 --2- 172.21.15.16:0/178485083 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fd9a002f2b0 0x7fd9a0031770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:56:32.714+0000 7fd9c6f70700 1 --2- 172.21.15.16:0/178485083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9c007f3a0 0x7fd9c0117fa0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-03-30T18:56:53.209+0000 7f52f0ff9700 1 -- 172.21.15.16:0/646304781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52e801d8e0 con 0x7f52f4065b10 , 2021-03-30T18:56:53.209+0000 7f52fafc9700 1 -- 172.21.15.16:0/646304781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f412b390 con 0x7f52f4065b10 , 2021-03-30T18:56:53.209+0000 7f52fafc9700 1 -- 172.21.15.16:0/646304781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f412b7a0 con 0x7f52f4065b10 Ignoring malparsed backtrace: 2021-03-30T18:57:13.870+0000 7fcde9ff3700 1 --2- 172.21.15.16:0/2637255144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcde411b890 0x7fcde411bcb0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:57:13.870+0000 7fcde9ff3700 1 -- 172.21.15.16:0/2637255144 >> 172.21.15.16:0/2637255144 conn(0x7fcde4062cf0 msgr2=0x7fcde4065110 unknown :-1 s=STATE_NONE l=0).mark_down , 2021-03-30T18:57:13.870+0000 7fcde9ff3700 1 -- 172.21.15.16:0/2637255144 shutdown_connections Ignoring malparsed backtrace: 2021-03-30T18:57:34.546+0000 7fc66eac2700 1 -- start start , 2021-03-30T18:57:34.546+0000 7fc66eac2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6680676b0 0x7fc66811fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:57:34.546+0000 7fc66eac2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66807e810 con 0x7fc668067c00 Ignoring malparsed backtrace: 2021-03-30T18:57:35.057+0000 7fec11ffe700 1 -- 172.21.15.16:0/2190954222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec0c081fc0 con 0x7fec0c0663a0 , 2021-03-30T18:57:35.058+0000 7febf37fe700 1 -- 172.21.15.16:0/2190954222 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+19110 (secure 0 0 0) 0x7febfc00fe60 con 0x7fec0c0663a0 , 2021-03-30T18:57:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2021-03-30T18:57:55.574+0000 7f4febcc0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4117790 con 0x7f4fe411b150 , 2021-03-30T18:57:55.574+0000 7f4fe9a5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fe411b150 0x7f4fe4089780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:57:55.574+0000 7f4fe9a5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fe411b150 0x7f4fe4089780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51504/0 (socket says 172.21.15.16:51504) Ignoring malparsed backtrace: 2021-03-30T18:58:16.237+0000 7f174dd83700 1 -- 172.21.15.16:0/1766276438 >> v1:172.21.15.16:6789/0 conn(0x7f17480663a0 legacy=0x7f1748066830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T18:58:16.237+0000 7f174dd83700 1 -- 172.21.15.16:0/1766276438 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174811acb0 con 0x7f174811b150 , 2021-03-30T18:58:16.237+0000 7f174cd81700 1 -- 172.21.15.16:0/1766276438 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301879355 0 0) 0x7f17480680e0 con 0x7f17480663a0 Ignoring malparsed backtrace: 2021-03-30T18:58:36.747+0000 7fc7ea873700 1 --2- 172.21.15.16:0/3563008885 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7fc7c402f2a0 0x7fc7c4031760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:58:36.747+0000 7fc7ea873700 1 --2- 172.21.15.16:0/3563008885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7e4065d70 0x7fc7e412d300 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop , 2021-03-30T18:58:36.747+0000 7fc7ea873700 1 -- 172.21.15.16:0/3563008885 >> 172.21.15.16:0/3563008885 conn(0x7fc7e40926f0 msgr2=0x7fc7e40654b0 unknown :-1 s=STATE_NONE l=0).mark_down Ignoring malparsed backtrace: 2021-03-30T18:58:57.262+0000 7f31eedc5700 1 -- 172.21.15.16:0/1355447012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e8066e90 con 0x7f31e80888f0 , 2021-03-30T18:58:57.263+0000 7f31e4ff9700 1 -- 172.21.15.16:0/1355447012 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 35601+0+0 (secure 0 0 0) 0x7f31dc00f070 con 0x7f31e80888f0 , 2021-03-30T18:58:57.263+0000 7f31e4ff9700 1 --2- 172.21.15.16:0/1355447012 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7f31c802f2b0 0x7f31c8031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect Ignoring malparsed backtrace: 2021-03-30T18:59:18.036+0000 7f8cfd667700 1 --2- 172.21.15.16:0/1541290232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf811dfb0 0x7f8cf81203a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7f8ce800b940 tx=0x7f8ce8006e20).stop , 2021-03-30T18:59:18.037+0000 7f8cfd667700 1 -- 172.21.15.16:0/1541290232 shutdown_connections , 2021-03-30T18:59:18.037+0000 7f8cfd667700 1 --2- 172.21.15.16:0/1541290232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf811dfb0 0x7f8cf81203a0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 rx=0 tx=0).stop Ignoring malparsed backtrace: 2021-03-30T18:59:18.667 INFO:tasks.ceph:Waiting for all PGs to be active+clean and split+merged, waiting on ['3.3f', '4.38', '3.3e', '4.39', '3.3d', '4.3a', '3.3c', '4.3b', '3.3b', '4.3c', '3.3a', '4.3d', '3.39', '4.3e', '3.38', '4.3f', '3.37', '4.30', '3.36', '4.31', '3.35', '4.32', '3.34', '4.33', '3.33', '4.34', '3.32', '4.35', '3.31', '4.36', '3.30', '4.37', '3.2f', '4.28', '3.2e', '4.29', '3.2d', '4.2a', '3.2c', '4.2b', '3.2b', '4.2c', '3.2a', '4.2d', '3.29', '4.2e', '3.28', '4.2f', '3.27', '4.20', '3.26', '4.21', '3.25', '4.22', '3.24', '4.23', '3.23', '4.24', '3.22', '4.25', '3.21', '4.26', '3.20', '4.27', '9.15', '8.14', '7.1b', '6.1a', '3.1f', '4.18', '5.19', '9.14', '8.15', '7.1a', '6.1b', '3.1e', '4.19', '5.18', '9.17', '8.16', '7.19', '6.18', '3.1d', '4.1a', '5.1b', '9.16', '8.17', '7.18', '6.19', '3.1c', '4.1b', '5.1a', '9.11', '8.10', '7.1f', '6.1e', '5.1d', '3.1b', '4.1c', '9.c', '8.d', '7.2', '6.3', '4.1', '3.6', '2.7', '5.0', '9.d', '8.c', '7.3', '6.2', '4.0', '2.6', '3.7', '5.1', '9.e', '8.f', '7.0', '6.1', '4.3', '3.4', '2.5', '5.2', '9.b', '8.a', '7.5', '6.4', '3.1', '2.0', '4.6', '5.7', '9.8', '8.9', '7.6', '6.7', '3.2', '1.0', '2.3', '4.5', '5.4', '9.a', '8.b', '7.4', '6.5', '3.0', '2.1', '4.7', '5.6', '9.9', '8.8', '7.7', '6.6', '4.4', '3.3', '2.2', '5.5', '9.f', '8.e', '7.1', '6.0', '4.2', '3.5', '2.4', '5.3', '9.2', '8.3', '7.c', '6.d', '3.8', '4.f', '5.e', '9.3', '8.2', '7.d', '6.c', '3.9', '4.e', '5.f', '9.0', '8.1', '7.e', '6.f', '3.a', '4.d', '5.c', '9.1', '8.0', '7.f', '6.e', '3.b', '4.c', '5.d', '9.6', '8.7', '7.8', '6.9', '3.c', '4.b', '5.a', '9.7', '8.6', '7.9', '6.8', '3.d', '4.a', '5.b', '9.4', '8.5', '7.a', '6.b', '3.e', '4.9', '5.8', '9.5', '8.4', '7.b', '6.a', '3.f', '4.8', '5.9', '9.1a', '8.1b', '7.14', '6.15', '3.10', '4.17', '5.16', '9.1b', '8.1a', '7.15', '6.14', '3.11', '4.16', '5.17', '9.18', '8.19', '7.16', '6.17', '3.12', '4.15', '5.14', '9.19', '8.18', '7.17', '6.16', '3.13', '4.14', '5.15', '9.1e', '8.1f', '7.10', '6.11', '3.14', '4.13', '5.12', '9.1f', '8.1e', '7.11', '6.10', '3.15', '4.12', '5.13', '9.1c', '8.1d', '7.12', '6.13', '3.16', '4.11', '5.10', '9.1d', '8.1c', '7.13', '6.12', '3.17', '4.10', '5.11', '9.12', '8.13', '7.1c', '6.1d', '5.1e', '3.18', '4.1f', '9.13', '8.12', '7.1d', '6.1c', '5.1f', '3.19', '4.1e', '9.10', '8.11', '7.1e', '6.1f', '3.1a', '4.1d', '5.1c'] to go clean , 2021-03-30T18:59:38.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-03-30T18:59:38.808+0000 7f82b7ae1700 1 Processor -- start Ignoring malparsed backtrace: 2021-03-30T18:59:39.213+0000 7ff533fff700 1 -- 172.21.15.16:0/3617875649 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5858+0+0 (secure 0 0 0) 0x7ff53c03f9b0 con 0x7ff54c066650 , 2021-03-30T18:59:39.213+0000 7ff54b7fe700 1 --2- 172.21.15.16:0/3617875649 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7ff52c02f2b0 0x7ff52c031770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 , 2021-03-30T18:59:39.213+0000 7ff54b7fe700 1 --2- 172.21.15.16:0/3617875649 >> [v2:172.21.15.16:6803/27507,v1:172.21.15.16:6805/27507] conn(0x7ff52c02f2b0 0x7ff52c031770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff54000b940 tx=0x7ff540016000).ready entity=mgr.4248 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2021-03-30T18:59:59.916+0000 7f8b2384d700 1 -- start start , 2021-03-30T18:59:59.916+0000 7f8b2384d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b1c065e90 0x7f8b1c0882a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect , 2021-03-30T18:59:59.916+0000 7f8b2384d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c087330 con 0x7f8b1c065e90 Ignoring malparsed backtrace: 2021-03-30T19:00:20.589+0000 7fcd1ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd2011c1a0 0x7fcd2007e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45014/0 (socket says 172.21.15.16:45014) , 2021-03-30T19:00:20.589+0000 7fcd1ffff700 1 -- 172.21.15.16:0/1833495911 learned_addr learned my addr 172.21.15.16:0/1833495911 (peer_addr_for_me v1:172.21.15.16:0/0) , 2021-03-30T19:00:20.590+0000 7fcd248dc700 1 -- 172.21.15.16:0/1833495911 >> v1:172.21.15.16:6789/0 conn(0x7fcd2011c1a0 legacy=0x7fcd2007e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2021-03-30T19:00:41.100+0000 7ff943a61700 1 -- 172.21.15.16:0/3921928641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff93c064930 msgr2=0x7ff93c120df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T19:00:41.100+0000 7ff943a61700 1 --2- 172.21.15.16:0/3921928641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff93c064930 0x7ff93c120df0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7ff93800b150 tx=0x7ff938016000).stop , 2021-03-30T19:00:41.100+0000 7ff943a61700 1 -- 172.21.15.16:0/3921928641 shutdown_connections Ignoring malparsed backtrace: 2021-03-30T19:01:01.592+0000 7fb1937fe700 1 -- 172.21.15.16:0/2384871957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb19c0054f0 con 0x7fb1ac0676b0 , 2021-03-30T19:01:01.592+0000 7fb1937fe700 1 -- 172.21.15.16:0/2384871957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c005650 con 0x7fb1ac0676b0 , 2021-03-30T19:01:01.592+0000 7fb1b0896700 1 -- 172.21.15.16:0/2384871957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac080630 con 0x7fb1ac0676b0 Ignoring malparsed backtrace: 2021-03-30T19:01:22.266+0000 7f687d59a700 1 -- 172.21.15.16:0/746905030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6870009bf0 con 0x7f688011b150 , 2021-03-30T19:01:22.266+0000 7f6884967700 1 -- 172.21.15.16:0/746905030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f688011b150 msgr2=0x7f688011b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2021-03-30T19:01:22.266+0000 7f6884967700 1 --2- 172.21.15.16:0/746905030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f688011b150 0x7f688011b530 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f687000b940 tx=0x7f6870006e20).stop Ignoring malparsed backtrace: 2021-03-30T19:01:22.790+0000 7f570b252700 1 -- 172.21.15.16:0/1125698364 wait complete. , 2021-03-30T19:01:22.805 INFO:tasks.ceph:Waiting for all PGs to be active+clean and split+merged, waiting on ['3.3f', '4.38', '3.3e', '4.39', '3.3d', '4.3a', '3.3c', '4.3b', '3.3b', '4.3c', '3.3a', '4.3d', '3.39', '4.3e', '3.38', '4.3f', '3.37', '4.30', '3.36', '4.31', '3.35', '4.32', '3.34', '4.33', '3.33', '4.34', '3.32', '4.35', '3.31', '4.36', '3.30', '4.37', '3.2f', '4.28', '3.2e', '4.29', '3.2d', '4.2a', '3.2c', '4.2b', '3.2b', '4.2c', '3.2a', '4.2d', '3.29', '4.2e', '3.28', '4.2f', '3.27', '4.20', '3.26', '4.21', '3.25', '4.22', '3.24', '4.23', '3.23', '4.24', '3.22', '4.25', '3.21', '4.26', '3.20', '4.27', '9.15', '8.14', '7.1b', '6.1a', '3.1f', '4.18', '5.19', '9.14', '8.15', '7.1a', '6.1b', '3.1e', '4.19', '5.18', '9.17', '8.16', '7.19', '6.18', '3.1d', '4.1a', '5.1b', '9.16', '8.17', '7.18', '6.19', '3.1c', '4.1b', '5.1a', '9.11', '8.10', '7.1f', '6.1e', '5.1d', '3.1b', '4.1c', '9.c', '8.d', '7.2', '6.3', '4.1', '3.6', '2.7', '5.0', '9.d', '8.c', '7.3', '6.2', '4.0', '2.6', '3.7', '5.1', '9.e', '8.f', '7.0', '6.1', '4.3', '3.4', '2.5', '5.2', '9.b', '8.a', '7.5', '6.4', '3.1', '2.0', '4.6', '5.7', '9.8', '8.9', '7.6', '6.7', '3.2', '1.0', '2.3', '4.5', '5.4', '9.a', '8.b', '7.4', '6.5', '3.0', '2.1', '4.7', '5.6', '9.9', '8.8', '7.7', '6.6', '4.4', '3.3', '2.2', '5.5', '9.f', '8.e', '7.1', '6.0', '4.2', '3.5', '2.4', '5.3', '9.2', '8.3', '7.c', '6.d', '3.8', '4.f', '5.e', '9.3', '8.2', '7.d', '6.c', '3.9', '4.e', '5.f', '9.0', '8.1', '7.e', '6.f', '3.a', '4.d', '5.c', '9.1', '8.0', '7.f', '6.e', '3.b', '4.c', '5.d', '9.6', '8.7', '7.8', '6.9', '3.c', '4.b', '5.a', '9.7', '8.6', '7.9', '6.8', '3.d', '4.a', '5.b', '9.4', '8.5', '7.a', '6.b', '3.e', '4.9', '5.8', '9.5', '8.4', '7.b', '6.a', '3.f', '4.8', '5.9', '9.1a', '8.1b', '7.14', '6.15', '3.10', '4.17', '5.16', '9.1b', '8.1a', '7.15', '6.14', '3.11', '4.16', '5.17', '9.18', '8.19', '7.16', '6.17', '3.12', '4.15', '5.14', '9.19', '8.18', '7.17', '6.16', '3.13', '4.14', '5.15', '9.1e', '8.1f', '7.10', '6.11', '3.14', '4.13', '5.12', '9.1f', '8.1e', '7.11', '6.10', '3.15', '4.12', '5.13', '9.1c', '8.1d', '7.12', '6.13', '3.16', '4.11', '5.10', '9.1d', '8.1c', '7.13', '6.12', '3.17', '4.10', '5.11', '9.12', '8.13', '7.1c', '6.1d', '5.1e', '3.18', '4.1f', '9.13', '8.12', '7.1d', '6.1c', '5.1f', '3.19', '4.1e', '9.10', '8.11', '7.1e', '6.1f', '3.1a', '4.1d', '5.1c'] to go clean , 2021-03-30T19:01:42.807 ERROR:teuthology.contextutil:Saw exception from nested tasks Ignoring malparsed backtrace: 2021-03-30T19:01:42.924 INFO:teuthology.misc:Shutting down mon daemons... , 2021-03-30T19:01:42.925 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-03-30T19:01:42.925 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/task/internal/__init__.py", line 463, in coredump , yield Not-understood crash indication Binary file /home/teuthworker/archive/dang-2021-03-30_07:32:10-rgw-wip-dang-zipper-user-distro-basic-smithi/6011473/teuthology.log matches Not-understood crash indication Found 7 distinct failure reasons Timeout 3h running clone.client.0/qa/workunits/rgw/test_rgw_orphan_list.sh 1 jobs: ['6011473'] suites: ['cluster', 'rgw/tools/{centos_latest', 'tasks}'] Failure: too many values to unpack (expected 1) 3 jobs: ['6011469', '6011521', '6011495'] suites intersection: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/rgw_cephadm'] suites union: ['frontend/civetweb', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/singleton/{all/rgw_cephadm', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: list index out of range 1 jobs: ['6011465'] suites: ['rgw/ignore-pg-availability'] Failure: rgw multisite test failures 1 jobs: ['6011488'] suites: ['frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] Failure: "2021-03-30T16:08:11.426424+0000 mon.a (mon.0) 164 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 1 jobs: ['6011532'] suites: ['frontend/civetweb', 'objectstore/filestore-xfs', 'overrides', 'rgw/singleton/{all/radosgw-admin', 'rgw_pool_type/replicated', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed on smithi093 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '^tempest.api.object_storage' --black-regex '.*test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota|.*test_container_acl_negative.ObjectACLsNegativeTest.*|.*test_container_services_negative.ContainerNegativeTest.test_create_container_metadata_.*|.*test_container_staticweb.StaticWebTest.test_web_index|.*test_container_staticweb.StaticWebTest.test_web_listing_css|.*test_container_synchronization.*|.*test_object_services.PublicObjectTest.test_access_public_container_object_without_using_creds'" 1 jobs: ['6011471'] suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] Failure: Command failed on smithi042 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 1 jobs: ['6011468'] suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-random-distro$/{centos_8}', 'tasks/{0-install', 'test_kafka}}']