Found 5 jobs Ignoring malparsed backtrace: ceph version 17.0.0-7227-g8a3448f6 (8a3448f6a968cb63b89cca7ec318a2af24419382) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5580f1bcf2fb] , 2: (KernelDevice::_aio_thread()+0x1275) [0x5580f27e2865] Ignoring malparsed backtrace: ceph version 17.0.0-7227-g8a3448f6 (8a3448f6a968cb63b89cca7ec318a2af24419382) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f14d7cf2fb] , 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x9df) [0x55f14dfa35ef] Ignoring malparsed backtrace: 2021-08-25T08:32:56.462+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6804 osd.2 since back 2021-08-25T08:31:31.705873+0000 front 2021-08-25T08:31:31.705912+0000 (oldest deadline 2021-08-25T08:31:56.406012+0000) , 2021-08-25T08:32:56.462+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6812 osd.3 since back 2021-08-25T08:31:07.299013+0000 front 2021-08-25T08:31:07.299196+0000 (oldest deadline 2021-08-25T08:31:33.199086+0000) , 2021-08-25T08:32:56.628+0000 7f568d57c700 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.66:13315 5.7s0 5:e84b5107:::1000000b5e1.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) Ignoring malparsed backtrace: 2021-08-25T08:33:18.515+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6825 osd.1 since back 2021-08-25T08:30:08.483839+0000 front 2021-08-25T08:30:08.484172+0000 (oldest deadline 2021-08-25T08:30:33.783976+0000) , 2021-08-25T08:33:18.515+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6804 osd.2 since back 2021-08-25T08:31:31.705873+0000 front 2021-08-25T08:31:31.705912+0000 (oldest deadline 2021-08-25T08:31:56.406012+0000) , 2021-08-25T08:33:18.515+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6812 osd.3 since back 2021-08-25T08:31:07.299013+0000 front 2021-08-25T08:31:07.299196+0000 (oldest deadline 2021-08-25T08:31:33.199086+0000) Ignoring malparsed backtrace: 2021-08-25T08:33:35.437+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6825 osd.1 since back 2021-08-25T08:30:08.483839+0000 front 2021-08-25T08:30:08.484172+0000 (oldest deadline 2021-08-25T08:30:33.783976+0000) , 2021-08-25T08:33:35.437+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6804 osd.2 since back 2021-08-25T08:31:31.705873+0000 front 2021-08-25T08:31:31.705912+0000 (oldest deadline 2021-08-25T08:31:56.406012+0000) , 2021-08-25T08:33:35.437+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6812 osd.3 since back 2021-08-25T08:31:07.299013+0000 front 2021-08-25T08:31:07.299196+0000 (oldest deadline 2021-08-25T08:31:33.199086+0000) Ignoring malparsed backtrace: 2021-08-25T08:33:51.868+0000 7f6dd0499700 -1 osd.4 104 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.90:9493 3.3 3:c2aee1ac:::10000005201.00000000:head [omap-set-header in=274b,omap-set-vals in=1162b,omap-rm-keys in=48b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:33:52.385+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6825 osd.1 since back 2021-08-25T08:30:08.483839+0000 front 2021-08-25T08:30:08.484172+0000 (oldest deadline 2021-08-25T08:30:33.783976+0000) , 2021-08-25T08:33:52.385+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6804 osd.2 since back 2021-08-25T08:31:31.705873+0000 front 2021-08-25T08:31:31.705912+0000 (oldest deadline 2021-08-25T08:31:56.406012+0000) Ignoring malparsed backtrace: 2021-08-25T08:34:08.879+0000 7f6dd0499700 -1 osd.4 104 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.90:9493 3.3 3:c2aee1ac:::10000005201.00000000:head [omap-set-header in=274b,omap-set-vals in=1162b,omap-rm-keys in=48b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:34:08.909+0000 7f568d57c700 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.66:13315 5.7s0 5:e84b5107:::1000000b5e1.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:34:09.229+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6825 osd.1 since back 2021-08-25T08:30:08.483839+0000 front 2021-08-25T08:30:08.484172+0000 (oldest deadline 2021-08-25T08:30:33.783976+0000) Ignoring malparsed backtrace: 2021-08-25T08:34:25.317+0000 7ffa0659d700 -1 osd.0 98 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9495 3.6 3:60a0c2c0:::10000004f76.00000000:head [omap-set-header in=274b,omap-set-vals in=29672b,omap-rm-keys in=1546b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:34:25.958+0000 7f6dd0499700 -1 osd.4 104 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.90:9493 3.3 3:c2aee1ac:::10000005201.00000000:head [omap-set-header in=274b,omap-set-vals in=1162b,omap-rm-keys in=48b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:34:26.002+0000 7f568d57c700 -1 osd.5 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.66:13315 5.7s0 5:e84b5107:::1000000b5e1.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) Ignoring malparsed backtrace: 2021-08-25T08:34:42.451+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6812 osd.3 since back 2021-08-25T08:31:07.299013+0000 front 2021-08-25T08:31:07.299196+0000 (oldest deadline 2021-08-25T08:31:33.199086+0000) , 2021-08-25T08:34:42.451+0000 7ffa0659d700 -1 osd.0 98 get_health_metrics reporting 103 slow ops, oldest is osd_op(mds.1.66:13307 5.3s0 5:cec2f2a3:::1000000b527.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:34:42.915+0000 7f6dd0499700 -1 osd.4 104 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.90:9493 3.3 3:c2aee1ac:::10000005201.00000000:head [omap-set-header in=274b,omap-set-vals in=1162b,omap-rm-keys in=48b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) Ignoring malparsed backtrace: 2021-08-25T08:34:59.526+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6804 osd.2 since back 2021-08-25T08:31:31.705873+0000 front 2021-08-25T08:31:31.705912+0000 (oldest deadline 2021-08-25T08:31:56.406012+0000) , 2021-08-25T08:34:59.526+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6812 osd.3 since back 2021-08-25T08:31:07.299013+0000 front 2021-08-25T08:31:07.299196+0000 (oldest deadline 2021-08-25T08:31:33.199086+0000) , 2021-08-25T08:34:59.527+0000 7ffa0659d700 -1 osd.0 98 get_health_metrics reporting 103 slow ops, oldest is osd_op(mds.1.66:13307 5.3s0 5:cec2f2a3:::1000000b527.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) Ignoring malparsed backtrace: 2021-08-25T08:35:15.953+0000 7f6dd0499700 -1 osd.4 105 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.90:9493 3.3 3:c2aee1ac:::10000005201.00000000:head [omap-set-header in=274b,omap-set-vals in=1162b,omap-rm-keys in=48b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:35:16.307+0000 7f568d57c700 -1 osd.5 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.66:13315 5.7s0 5:e84b5107:::1000000b5e1.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:35:16.533+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6825 osd.1 since back 2021-08-25T08:30:08.483839+0000 front 2021-08-25T08:30:08.484172+0000 (oldest deadline 2021-08-25T08:30:33.783976+0000) Ignoring malparsed backtrace: 2021-08-25T08:35:32.433+0000 7ffa0659d700 -1 osd.0 98 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9495 3.6 3:60a0c2c0:::10000004f76.00000000:head [omap-set-header in=274b,omap-set-vals in=29672b,omap-rm-keys in=1546b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:35:33.082+0000 7f6dd0499700 -1 osd.4 105 get_health_metrics reporting 88 slow ops, oldest is osd_op(mds.0.90:9493 3.3 3:c2aee1ac:::10000005201.00000000:head [omap-set-header in=274b,omap-set-vals in=1162b,omap-rm-keys in=48b] snapc 0=[] ondisk+write+known_if_redirected+full_force e98) , 2021-08-25T08:35:33.377+0000 7f568d57c700 -1 osd.5 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.66:13315 5.7s0 5:e84b5107:::1000000b5e1.00000000:head [delete] snapc 278=[278] ondisk+write+known_if_redirected+full_force e98) Ignoring malparsed backtrace: 2021-08-25T08:35:49.398+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6825 osd.1 since back 2021-08-25T08:30:08.483839+0000 front 2021-08-25T08:30:08.484172+0000 (oldest deadline 2021-08-25T08:30:33.783976+0000) , 2021-08-25T08:35:49.398+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6804 osd.2 since back 2021-08-25T08:31:31.705873+0000 front 2021-08-25T08:31:31.705912+0000 (oldest deadline 2021-08-25T08:31:56.406012+0000) , 2021-08-25T08:35:49.398+0000 7ffa0659d700 -1 osd.0 98 heartbeat_check: no reply from 172.21.15.145:6812 osd.3 since back 2021-08-25T08:31:07.299013+0000 front 2021-08-25T08:31:07.299196+0000 (oldest deadline 2021-08-25T08:31:33.199086+0000) Ignoring malparsed backtrace: ceph version 17.0.0-7227-g8a3448f6 (8a3448f6a968cb63b89cca7ec318a2af24419382) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5595365cf2fb] , 2: (KernelDevice::_aio_thread()+0x1275) [0x5595371e2865] Ignoring malparsed backtrace: 2021-08-25T08:42:12.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T08:42:12.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T08:42:43.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T08:44:02.682+0000 7f568d57c700 -1 osd.5 118 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:44:03.124+0000 7f89944ae700 -1 osd.6 118 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:44:03.320+0000 7f6dd0499700 -1 osd.4 118 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:44:35.801+0000 7f6dd0499700 -1 osd.4 118 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:44:35.903+0000 7f89944ae700 -1 osd.6 118 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:44:36.642+0000 7f568d57c700 -1 osd.5 118 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:45:08.171+0000 7f568d57c700 -1 osd.5 119 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:45:08.174+0000 7f89944ae700 -1 osd.6 119 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:45:09.126+0000 7f89944ae700 -1 osd.6 119 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:45:51.952+0000 7f568d57c700 -1 osd.5 120 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:45:52.620+0000 7f89944ae700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:45:52.996+0000 7f568d57c700 -1 osd.5 120 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:46:40.734+0000 7f568d57c700 -1 osd.5 122 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:46:40.972+0000 7f89944ae700 -1 osd.6 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:46:41.215+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T08:47:03.910+0000 7f568d57c700 -1 osd.5 122 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:47:04.862+0000 7f89944ae700 -1 osd.6 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:47:04.897+0000 7f568d57c700 -1 osd.5 122 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:47:51.394+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:47:52.188+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:47:52.392+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:48:27.178+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:48:27.369+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:48:27.596+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:48:59.162+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:48:59.421+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:49:00.119+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:49:31.634+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:49:32.087+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:49:32.193+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:50:04.082+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:50:04.350+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:50:04.465+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:50:36.247+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:50:36.543+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:50:37.016+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:51:08.428+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:51:08.988+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:51:09.122+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:51:40.958+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:51:41.049+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:51:41.391+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:52:12.750+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:52:13.160+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:52:13.737+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:52:45.203+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:52:45.373+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:52:45.843+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:53:17.497+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:53:18.121+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:53:18.124+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:53:49.192+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:53:49.727+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:53:50.178+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:54:21.938+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:54:22.415+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:54:22.498+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:54:54.334+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:54:54.569+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:54:54.986+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:55:26.600+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:55:26.754+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T08:55:27.015+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:55:58.622+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:55:58.728+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:55:59.566+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:56:30.749+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:56:31.472+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:56:31.532+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:57:03.447+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:57:03.472+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:57:03.667+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:57:35.485+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:57:35.629+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:57:36.256+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:58:07.634+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:58:08.540+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:58:08.592+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:58:40.428+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:58:40.476+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:58:40.601+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:59:12.538+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:59:12.587+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:59:12.777+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T08:59:44.438+0000 7f89944ae700 -1 osd.6 124 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:59:44.657+0000 7f6dd0499700 -1 osd.4 124 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T08:59:45.343+0000 7f568d57c700 -1 osd.5 124 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:00:15.655+0000 7f6dd0499700 -1 osd.4 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:00:16.132+0000 7f568d57c700 -1 osd.5 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:00:16.413+0000 7f89944ae700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:00:46.862+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:00:46.956+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:00:47.416+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:01:19.000+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:01:19.406+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:01:19.965+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:01:50.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T09:01:51.057+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:01:51.078+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:02:23.056+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:02:23.147+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:02:23.829+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:02:55.150+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:02:55.937+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:02:55.964+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:03:27.406+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:03:27.922+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:03:28.062+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:04:00.123+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:04:00.147+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:04:00.416+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:04:32.250+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:04:32.566+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:04:33.143+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:05:04.421+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:05:05.231+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:05:05.348+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:05:37.006+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:05:37.441+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:05:37.447+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:06:09.595+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:06:09.682+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:06:09.794+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:06:39.795+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:06:39.805+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:06:39.969+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:07:12.478+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T09:07:13.092+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:07:13.178+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:07:46.337+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:07:46.410+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:07:46.669+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:08:19.154+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:08:19.785+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:08:19.827+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:08:51.823+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:08:52.181+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:08:52.422+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:09:24.330+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:09:24.358+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:09:24.969+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:09:56.503+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:09:57.119+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:09:57.158+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:10:28.402+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:10:28.639+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:10:29.348+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:11:00.510+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:11:01.069+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:11:01.470+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:11:33.237+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:11:33.517+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:11:33.528+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:12:05.389+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:12:05.403+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:12:06.266+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:12:37.539+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:12:38.320+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:12:38.463+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:13:10.464+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:13:10.464+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:13:10.597+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:13:42.612+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:13:42.752+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:13:42.884+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T09:14:14.688+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:14:14.700+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:14:15.392+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:14:46.761+0000 7f89944ae700 -1 osd.6 126 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:14:47.166+0000 7f568d57c700 -1 osd.5 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:14:47.653+0000 7f6dd0499700 -1 osd.4 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:15:24.431+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:15:24.720+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:15:25.429+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:16:12.382+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:16:12.674+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:16:13.031+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T09:16:58.789+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:16:58.981+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:16:59.795+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:17:46.574+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:17:46.848+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:17:47.570+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:18:23.984+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:18:24.035+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:18:24.319+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:18:56.138+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:18:56.336+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:18:56.954+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:19:28.294+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:19:28.403+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:19:29.017+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:20:00.161+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:20:00.766+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:20:01.083+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:20:32.657+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:20:32.938+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:20:33.234+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:21:05.061+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:21:05.207+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:21:06.003+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:21:37.213+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:21:37.963+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:21:38.096+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:22:09.925+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:22:10.026+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:22:10.227+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:22:42.227+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:22:42.310+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:22:42.951+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:23:14.382+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:23:14.403+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:23:15.032+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:23:46.466+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:23:47.186+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:23:47.334+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:24:19.280+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:24:19.423+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:24:19.424+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:24:51.535+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:24:51.815+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:24:52.055+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:25:23.806+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:25:24.178+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:25:24.370+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:25:55.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T09:25:56.104+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:25:56.126+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:26:28.135+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:26:28.160+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:26:28.256+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:26:59.900+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:27:00.300+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:27:00.548+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:27:32.051+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:27:32.695+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:27:32.807+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:28:04.693+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:28:04.939+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:28:05.185+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:28:36.968+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:28:37.376+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:28:37.735+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:29:09.164+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:29:09.847+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:29:10.131+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:29:41.861+0000 7f6dd0499700 -1 osd.4 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:29:41.955+0000 7f568d57c700 -1 osd.5 127 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:29:42.004+0000 7f89944ae700 -1 osd.6 127 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:30:11.847+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:30:12.734+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:30:12.861+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:30:43.865+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T09:30:44.651+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:30:44.738+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:31:16.593+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:31:16.800+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:31:17.151+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:31:48.790+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:31:49.459+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:31:49.762+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:32:21.471+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:32:21.640+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:32:21.703+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:32:53.766+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:32:53.830+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:32:54.370+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:33:26.359+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:33:26.415+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:33:26.685+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:33:57.585+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:33:58.487+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:33:58.493+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:34:29.813+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:34:30.489+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:34:30.503+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:35:02.480+0000 7f6dd0499700 -1 osd.4 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:35:02.584+0000 7f568d57c700 -1 osd.5 128 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:35:02.854+0000 7f89944ae700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:35:34.840+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:35:34.894+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:35:35.650+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:36:07.101+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:36:07.759+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:36:07.801+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:36:39.901+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:36:39.976+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:36:40.092+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:37:12.285+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:37:12.318+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:37:13.007+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:37:44.302+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:37:44.446+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:37:44.876+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:38:16.330+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:38:16.780+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:38:17.152+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:38:49.091+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:38:49.175+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:38:49.254+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:39:20.958+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:39:21.127+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:39:21.710+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:39:53.241+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:39:53.885+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:39:53.952+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:40:25.200+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:40:25.775+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:40:25.805+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:40:57.604+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:40:57.763+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:40:57.813+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:41:29.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T09:41:29.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T09:41:29.734+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:42:00.890+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:42:01.758+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:42:01.808+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:42:33.644+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:42:33.802+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:42:33.936+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:43:05.880+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:43:06.073+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:43:06.843+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:43:37.840+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:43:38.527+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:43:38.654+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:44:10.274+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:44:10.709+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:44:11.035+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:44:42.694+0000 7f6dd0499700 -1 osd.4 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:44:42.898+0000 7f89944ae700 -1 osd.6 129 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:44:43.645+0000 7f568d57c700 -1 osd.5 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:45:13.797+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:45:13.910+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:45:14.734+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T09:45:45.907+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:45:46.268+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:45:46.428+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:46:18.380+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:46:18.675+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:46:18.817+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:46:50.751+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:46:51.186+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:46:51.515+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:47:23.627+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:47:24.003+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:47:24.420+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:47:56.120+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:47:56.556+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:47:57.061+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:48:28.576+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:48:28.938+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:48:29.110+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:49:00.484+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:49:00.945+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:49:01.038+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:49:31.975+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:49:32.629+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:49:32.900+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:50:04.144+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:50:04.718+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:50:05.038+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:50:36.597+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:50:37.015+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:50:37.187+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:51:08.907+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:51:09.038+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:51:09.751+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:51:41.017+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:51:41.758+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:51:41.760+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:52:13.900+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:52:14.065+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:52:14.662+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:52:45.775+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:52:46.322+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:52:46.683+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:53:18.475+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:53:18.850+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:53:18.990+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:53:50.783+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:53:51.276+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:53:51.348+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:54:23.268+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:54:23.354+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:54:23.791+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:54:55.307+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:54:55.877+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:54:56.201+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:55:27.743+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:55:28.022+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:55:28.239+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:55:59.984+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:56:00.080+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:56:00.410+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T09:56:32.053+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:56:32.100+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:56:32.858+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:57:04.205+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:57:04.281 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T09:57:04.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T09:57:36.237+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:57:36.437+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:57:36.652+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:58:08.586+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:58:08.701+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:58:09.182+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:58:40.912+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:58:41.017+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:58:41.744+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:59:12.927+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:59:13.701+0000 7f6dd0499700 -1 osd.4 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:59:13.828+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T09:59:45.639+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T09:59:45.678+0000 7f568d57c700 -1 osd.5 130 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T09:59:45.950+0000 7f89944ae700 -1 osd.6 130 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:00:15.791+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:00:16.139+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:00:16.705+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:00:48.380+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:00:48.509+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:00:48.958+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:01:20.601+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:01:20.683+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:01:21.510+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:01:52.745+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:01:53.548+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:01:53.679+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:02:25.533+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:02:25.958+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:02:26.059+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:02:57.849+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:02:57.977+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:02:58.470+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:03:29.884+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:03:30.688+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:03:30.801+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:04:01.948+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:04:02.677+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:04:02.785+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:04:34.719+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:04:34.761+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:04:34.896+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:05:06.686+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:05:06.790+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:05:06.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T10:05:38.355+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:05:38.458+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:05:38.568+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:06:10.330+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:06:10.686+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:06:11.295+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:06:42.749+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:06:43.320+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:06:43.498+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:07:15.128+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:07:15.695+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:07:15.723+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:07:47.040+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:07:47.704+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:07:47.775+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:08:19.715+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:08:19.803+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:08:19.967+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:08:51.857+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:08:51.895+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:08:52.819+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:09:23.974+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:09:24.576+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:09:24.899+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:09:56.364+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:09:57.008+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:09:57.040+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:10:28.318+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:10:28.808+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:10:29.223+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:11:00.855+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:11:00.982+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:11:01.187+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:11:32.694+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:11:33.054+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:11:33.137+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:12:04.937+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:12:05.072+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:12:05.925+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:12:37.133+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:12:37.774+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:12:38.111+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:13:09.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T10:13:09.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T10:13:09.776+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:13:40.910+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:13:41.916+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:13:41.921+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:14:13.878+0000 7f568d57c700 -1 osd.5 131 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:14:13.879+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:14:14.064+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:14:46.002+0000 7f6dd0499700 -1 osd.4 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:14:46.094+0000 7f89944ae700 -1 osd.6 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:14:46.549+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T10:15:16.845+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:15:16.898+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:15:17.183+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:15:48.830+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:15:49.335+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:15:49.571+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:16:21.497+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:16:21.844+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:16:21.885+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:16:53.787+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:16:54.275+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:16:54.587+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:17:26.108+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:17:26.840+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:17:26.990+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:17:58.848+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:17:58.996+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:17:59.282+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:18:31.006+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:18:31.058+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:18:31.762+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T10:19:03.015+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:19:03.294+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:19:03.891+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:19:35.395+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:19:36.046+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:19:36.073+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:20:07.239+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:20:07.875+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:20:08.053+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:20:39.789+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:20:39.903+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:20:40.026+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:21:11.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T10:21:11.639+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:21:11.842+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:21:43.458+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:21:43.692+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:21:43.810+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:22:15.498+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:22:15.778+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:22:16.159+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:22:47.725+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:22:47.730+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:22:48.075+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:23:19.695+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:23:19.981+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:23:20.525+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:23:52.168+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:23:52.505+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:23:52.533+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:24:24.458+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:24:24.755+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:24:24.847+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:24:56.811+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:24:56.841+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:24:57.371+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:25:29.077+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:25:29.398+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:25:29.916+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:26:01.502+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:26:02.262+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T10:26:02.276+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:26:33.714+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:26:34.263+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:26:34.472+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:27:06.161+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:27:06.354+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:27:06.524+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:27:38.507+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:27:38.803+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:27:39.307+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:28:10.850+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:28:11.429+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:28:11.555+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:28:43.682+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:28:43.701+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:28:43.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T10:29:14.902+0000 7f89944ae700 -1 osd.6 132 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:29:15.511+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:29:15.590+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:29:47.432+0000 7f568d57c700 -1 osd.5 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:29:47.497+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T10:29:47.639+0000 7f6dd0499700 -1 osd.4 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:30:17.984+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:30:18.271+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:30:18.787+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:30:50.232+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:30:50.892+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:30:51.221+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:31:23.095+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:31:23.132+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:31:23.180+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:31:55.250+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:31:55.392+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:31:55.850+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:32:27.568+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T10:32:28.029+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:32:28.307+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:33:00.129+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:33:00.388+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:33:00.694+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T10:33:32.260+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:33:32.666+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:33:32.720+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T10:34:04.176+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:34:04.780+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:34:04.968+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:34:36.833+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:34:37.014+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:34:37.336+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:35:09.208+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:35:09.325+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:35:09.841+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:35:41.996+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:35:42.191+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:35:42.448+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:36:14.183+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:36:14.296+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:36:15.093+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:36:46.245+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:36:46.336+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:36:46.395+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:37:18.311+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:37:18.495+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:37:18.528+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:37:50.440+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:37:50.498+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:37:51.436+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:38:22.685+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:38:23.251+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:38:23.494+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:38:55.178+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:38:55.595+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:38:55.908+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:39:27.452+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:39:27.832+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:39:28.243+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:39:59.798+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:39:59.999+0000 7f29c7ecd700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; 1 osds down; 1 host (4 osds) down; Reduced data availability: 8 pgs inactive, 1 pg down, 7 pgs stale; Degraded data redundancy: 49367/153330 objects degraded (32.197%), 11 pgs degraded, 16 pgs undersized; 69 slow ops, oldest one blocked for 7159 sec, daemons [osd.0,osd.4,osd.5,osd.6] have slow ops. , 2021-08-25T10:40:00.341+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:40:31.947+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:40:32.305+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:40:32.669+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:41:03.806+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:41:04.287+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:41:04.712+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:41:36.172+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:41:36.681+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:41:37.057+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:42:08.487+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:42:09.088+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:42:09.166+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:42:40.931+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:42:41.248+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:42:41.534+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:43:13.503+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:43:13.632+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:43:13.787+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:43:45.625+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:43:45.783+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:43:46.508+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:44:17.637+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:44:18.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T10:44:18.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T10:44:49.078+0000 7f568d57c700 -1 osd.5 133 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:44:49.435+0000 7f6dd0499700 -1 osd.4 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:44:49.637+0000 7f89944ae700 -1 osd.6 133 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:45:27.387+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:45:27.644+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:45:28.350+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:46:15.409+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:46:15.727+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:46:16.425+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:47:02.302+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:47:02.528+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:47:03.305+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:47:49.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T10:47:49.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T10:47:49.488+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:48:27.261+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:48:27.269+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:48:27.379+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:48:59.226+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:48:59.345+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:49:00.241+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:49:31.370+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:49:32.063+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:49:32.357+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:50:03.598+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T10:50:03.625+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:50:04.142+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:50:35.737+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:50:35.920+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:50:35.968+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:51:07.909+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:51:07.913+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:51:08.909+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:51:39.992+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:51:40.838+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:51:40.943+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:52:12.675+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:52:12.822+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:52:13.201+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:52:45.426+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:52:46.085+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:52:46.228+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:53:18.039+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:53:18.315+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:53:18.431+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:53:49.950+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:53:50.439+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:53:50.678+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:54:21.781+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:54:21.888+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:54:22.734+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:54:54.331+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:54:54.468+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:54:54.806+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:55:26.729+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:55:27.213+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:55:27.270+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:55:58.911+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:55:59.118+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:55:59.797+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:56:31.173+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:56:31.546+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:56:31.833+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:57:03.282+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:57:03.885+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:57:03.936+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T10:57:35.468+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:57:35.797+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:57:36.294+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:58:07.693+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:58:08.175+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:58:08.397+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:58:40.353+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:58:40.582+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:58:40.596+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:59:12.678+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:59:12.843+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:59:13.663+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T10:59:45.121+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:59:45.654+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T10:59:45.722+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:00:16.138+0000 7f89944ae700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:00:16.427+0000 7f568d57c700 -1 osd.5 134 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:00:16.885+0000 7f6dd0499700 -1 osd.4 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:00:48.402+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:00:48.886+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:00:49.017+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:01:21.033+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:01:21.174+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:01:21.444+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:01:53.317+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:01:53.488+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:01:53.829 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T11:02:24.804+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:02:25.186+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:02:25.697+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:02:57.140+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:02:57.481+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:02:57.660+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:03:29.738+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:03:29.778+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:03:29.879+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:04:01.644+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:04:01.959+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:04:02.586+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:04:34.225+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:04:34.378+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T11:04:34.521+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:05:06.227+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:05:06.288+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:05:06.339+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:05:38.288+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:05:38.545+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:05:39.037+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:06:10.673+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:06:11.112+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:06:11.346+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:06:43.128+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:06:43.163+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:06:43.530+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:07:15.404+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:07:15.466+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:07:16.264+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:07:47.666+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:07:48.032+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:07:48.481+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:08:19.884+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:08:19.920+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:08:20.649+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:08:52.026+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:08:52.610+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:08:53.005+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:09:24.793+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:09:24.950+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:09:25.233+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:09:56.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T11:09:56.777+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:09:57.032+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:10:27.918+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:10:28.481+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:10:28.734+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:11:00.185+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:11:00.768+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:11:00.833+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:11:32.579+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:11:32.958+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:11:33.443+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:12:04.841+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:12:04.852+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T11:12:05.509+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:12:36.776+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:12:37.472+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:12:37.733+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:13:09.629+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:13:09.745+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:13:09.765+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:13:41.913+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:13:42.069+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:13:42.758+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:14:14.183+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:14:14.958+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:14:15.089+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:14:47.070+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:14:47.088+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:14:47.246+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:15:19.162+0000 7f6dd0499700 -1 osd.4 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:15:19.405+0000 7f568d57c700 -1 osd.5 135 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:15:20.020+0000 7f89944ae700 -1 osd.6 135 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:15:50.187+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:15:50.289+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:15:51.098+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:16:23.220+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:16:23.397+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:16:23.470+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:16:55.468+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:16:55.581+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:16:56.004+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:17:27.708+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:17:27.925+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:17:28.504+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:17:59.715+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:17:59.779+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:18:00.102+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:18:31.914+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:18:32.085+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:18:32.321+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:19:04.253+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:19:04.485+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:19:04.811+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:19:36.250+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:19:36.679+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:19:36.853+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:20:08.475+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:20:09.049+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:20:09.101+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:20:40.813+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:20:41.028+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:20:41.263+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:21:12.784+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:21:13.104+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:21:13.647+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:21:45.119+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:21:45.547+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:21:45.889+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:22:17.822+0000 7f89944ae700 -1 osd.6 136 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:22:17.848+0000 7f568d57c700 -1 osd.5 136 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:22:17.988+0000 7f6dd0499700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:22:49.884+0000 7f568d57c700 -1 osd.5 138 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:22:50.150+0000 7f6dd0499700 -1 osd.4 138 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:22:50.468+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T11:23:22.039+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:23:22.047+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:23:22.835+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:23:53.970+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:23:53.986+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:23:54.919+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:24:26.088+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:24:26.932+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:24:27.029+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:24:58.290+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:24:59.033+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:24:59.234+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:25:29.338+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:25:30.162+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:25:30.179+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:25:42.285 INFO:tasks.mds_thrash:joining mds_thrasher , 2021-08-25T11:25:42.287 ERROR:teuthology.run_tasks:Manager failed: mds_thrash , Traceback (most recent call last): Ignoring malparsed backtrace: 2021-08-25T11:26:10.434+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:26:10.556+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:26:10.970+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:26:43.472+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:26:43.771+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:26:43.936+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:27:16.442+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:27:16.990+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:27:17.220+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:27:49.007+0000 7f6dd0499700 -1 osd.4 140 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:27:49.088+0000 7f568d57c700 -1 osd.5 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:27:49.498+0000 7f89944ae700 -1 osd.6 140 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:28:20.895+0000 7f6dd0499700 -1 osd.4 142 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:28:21.148+0000 7f568d57c700 -1 osd.5 142 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:28:21.571+0000 7f89944ae700 -1 osd.6 142 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:28:53.142+0000 7f568d57c700 -1 osd.5 142 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:28:53.179+0000 7f89944ae700 -1 osd.6 142 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:28:53.806+0000 7f6dd0499700 -1 osd.4 142 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:29:25.125+0000 7f568d57c700 -1 osd.5 142 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:29:25.941+0000 7f6dd0499700 -1 osd.4 142 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:29:25.992+0000 7f89944ae700 -1 osd.6 142 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:29:57.795+0000 7f6dd0499700 -1 osd.4 144 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:29:58.101+0000 7f89944ae700 -1 osd.6 144 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:29:58.223+0000 7f568d57c700 -1 osd.5 144 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:30:31.155+0000 7f568d57c700 -1 osd.5 145 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:30:32.059+0000 7f89944ae700 -1 osd.6 145 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:30:32.188+0000 7f568d57c700 -1 osd.5 145 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:31:18.367+0000 7f89944ae700 -1 osd.6 147 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:31:18.764+0000 7f568d57c700 -1 osd.5 147 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:31:19.398+0000 7f89944ae700 -1 osd.6 147 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:32:06.036+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T11:32:06.402+0000 7f89944ae700 -1 osd.6 147 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:32:06.450+0000 7f568d57c700 -1 osd.5 147 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:32:52.530+0000 7f89944ae700 -1 osd.6 147 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:32:53.377+0000 7f568d57c700 -1 osd.5 147 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:32:53.533+0000 7f89944ae700 -1 osd.6 147 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:33:35.486+0000 7f568d57c700 -1 osd.5 147 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:33:35.543+0000 7f89944ae700 -1 osd.6 147 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:33:35.617+0000 7f6dd0499700 -1 osd.4 147 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:34:07.405+0000 7f568d57c700 -1 osd.5 149 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:34:07.590+0000 7f6dd0499700 -1 osd.4 149 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:34:07.928+0000 7f89944ae700 -1 osd.6 149 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:34:38.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T11:34:39.531+0000 7f568d57c700 -1 osd.5 149 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:34:39.631+0000 7f89944ae700 -1 osd.6 149 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:35:11.489+0000 7f568d57c700 -1 osd.5 149 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:35:11.701+0000 7f6dd0499700 -1 osd.4 149 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:35:11.746+0000 7f89944ae700 -1 osd.6 149 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:35:43.666+0000 7f89944ae700 -1 osd.6 149 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:35:43.772+0000 7f6dd0499700 -1 osd.4 149 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:35:44.451+0000 7f568d57c700 -1 osd.5 149 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:36:15.894+0000 7f89944ae700 -1 osd.6 149 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:36:16.243+0000 7f568d57c700 -1 osd.5 149 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:36:16.840+0000 7f6dd0499700 -1 osd.4 149 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:36:48.668+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:36:48.851+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:36:49.144+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:37:20.966+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:37:21.068+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:37:21.355+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T11:37:52.671+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:37:52.676+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:37:53.246+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:38:24.669+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:38:25.247+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:38:25.334+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:38:57.124+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:38:57.369+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:38:57.550+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:39:29.493+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:39:29.772+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:39:30.211+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:40:01.984+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:40:02.079+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:40:02.228+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:40:34.019+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:40:34.036+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:40:34.795+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:40:49.847+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:40:49.963+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:40:50.799+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:41:21.858+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:41:22.122+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:41:22.755+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:41:54.167+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:41:54.672+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:41:54.808+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:42:26.902+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:42:26.926+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:42:27.005+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:42:59.143+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:42:59.180+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:43:00.028+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:43:31.347+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:43:32.026+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:43:32.192+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:44:03.723+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:44:04.236+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:44:04.388+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:44:36.338+0000 7f89944ae700 -1 osd.6 151 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:44:36.445+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:44:36.792+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T11:45:08.370+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:45:08.689+0000 7f568d57c700 -1 osd.5 151 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:45:09.321+0000 7f6dd0499700 -1 osd.4 151 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:45:39.973+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:45:39.980+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:45:40.501+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:46:12.129+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:46:12.162+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:46:12.346 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T11:46:44.038+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:46:44.045+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:46:44.055+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:47:15.853+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:47:16.009+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:47:16.823+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:47:47.955+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:47:48.631+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:47:48.942+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:48:20.822+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:48:21.038+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:48:21.066+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:48:52.663+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:48:53.327+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:48:53.402+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:49:25.302+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:49:25.410+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:49:25.449+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:49:57.482+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:49:57.487+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:49:58.205+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:50:29.589+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:50:29.744+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:50:29.978+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:51:01.685+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:51:01.794+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:51:02.646+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:51:33.711+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:51:34.378+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:51:34.552+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:52:06.347+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:52:06.591+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:52:06.975+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:52:38.173+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:52:38.485+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:52:39.171+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:53:10.508+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:53:11.197+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:53:11.476+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:53:43.276+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:53:43.281+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:53:43.606+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:54:14.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T11:54:15.126+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:54:15.283+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:54:47.051+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:54:47.292+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:54:47.490+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:55:19.461+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:55:19.479+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:55:20.011+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:55:51.497+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:55:51.860+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:55:52.377+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:56:23.486+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:56:23.845+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:56:24.382+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:56:56.231+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:56:56.287+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:56:56.403+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:57:28.075+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:57:28.148+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:57:29.011+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:58:00.006+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:58:00.833+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:58:00.857+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:58:32.787+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:58:32.901+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:58:33.028+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:59:04.942+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:59:04.954+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:59:05.527+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T11:59:37.177+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:59:37.199+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T11:59:37.692+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T12:00:08.903+0000 7f568d57c700 -1 osd.5 152 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:00:09.266+0000 7f6dd0499700 -1 osd.4 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:00:09.393+0000 7f89944ae700 -1 osd.6 152 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:00:40.350+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:00:40.350+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:00:40.987+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:01:12.358+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:01:13.245+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:01:13.337+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:01:45.157+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:01:45.320+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:01:45.591+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:02:17.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T12:02:17.179+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:02:17.253+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:02:49.188+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:02:49.281+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:02:49.391+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:03:21.282+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:03:21.383+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:03:22.097+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:03:53.034+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:03:53.263+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:03:53.938+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:04:25.157+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:04:25.864+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:04:25.960+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:04:57.850+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:04:57.895+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:04:57.927+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:05:29.807+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:05:29.857+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:05:30.665+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:06:02.014+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:06:02.603+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:06:02.665+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:06:34.768+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:06:34.842+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:06:34.933+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:07:07.014+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:07:07.055+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:07:07.803+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:07:39.153+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:07:39.245+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:07:39.846+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:08:11.392+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:08:11.734+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:08:12.262+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:08:43.887+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:08:44.513+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:08:44.544+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:09:16.352+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:09:16.582+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:09:16.932+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:09:48.561+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:09:49.000+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:09:49.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T12:10:20.079+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:10:20.436+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:10:20.461+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:10:52.347+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:10:52.355+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:10:53.152+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:11:24.183+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:11:24.349+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:11:25.007+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:11:56.138+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:11:57.075+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:11:57.125+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:12:28.909+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:12:29.141+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:12:29.260+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:13:01.030+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:13:01.359+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:13:02.050+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:13:33.233+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:13:33.751+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:13:34.002+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:14:05.918+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:14:06.249+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:14:06.336+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:14:38.064+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:14:38.585+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:14:38.596+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T12:15:10.121+0000 7f89944ae700 -1 osd.6 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:15:10.716+0000 7f568d57c700 -1 osd.5 153 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:15:10.732+0000 7f6dd0499700 -1 osd.4 153 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:15:50.264+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:15:50.707+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:15:50.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T12:16:36.837+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:16:37.664+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:16:37.884+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:17:23.754+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T12:17:23.875+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:17:24.346+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:18:11.242+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:18:11.796+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:18:12.228+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:18:47.951+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:18:47.968+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:18:48.056+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:19:20.118+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:19:20.256+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:19:20.863+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:19:52.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T12:19:52.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T12:19:52.391+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:20:23.554+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:20:23.936+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T12:20:24.077+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:20:55.756+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:20:56.009+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:20:56.305+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:21:28.062+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:21:28.086+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:21:28.856+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:22:00.315+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:22:00.833+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:22:01.088+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:22:32.812+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:22:33.251+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:22:33.645+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:23:04.921+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:23:05.751+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:23:05.895+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:23:37.664+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:23:37.731+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:23:37.940+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:24:09.616+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:24:09.704+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:24:10.359+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:24:41.797+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:24:42.303+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:24:42.514+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:25:14.420+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:25:14.481+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:25:14.828+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:25:46.718+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:25:47.088+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:25:47.636+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:26:19.170+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:26:19.501+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:26:19.844+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:26:51.464+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:26:51.639+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:26:52.286+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:27:23.624+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:27:24.256+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:27:24.300 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T12:27:55.320+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:27:55.635+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:27:55.853+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:28:27.678+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:28:28.054+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:28:28.353+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:29:00.083+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:29:00.245+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:29:00.372+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:29:32.139+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:29:32.205+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:29:33.131+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:30:04.094+0000 7f89944ae700 -1 osd.6 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:30:04.209+0000 7f568d57c700 -1 osd.5 154 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:30:05.037+0000 7f6dd0499700 -1 osd.4 154 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:30:35.199+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T12:30:35.331+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:30:36.105+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:31:07.381+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:31:08.222+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T12:31:08.248+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:31:39.587+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T12:31:39.945+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T12:31:40.498+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:32:12.101+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T12:32:12.459+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:32:12.466+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:32:44.250+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:32:44.875+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:32:44.942+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) Ignoring malparsed backtrace: 2021-08-25T12:33:16.697+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:33:16.988+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T12:33:17.538+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:33:49.067+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:33:49.579+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:33:49.610+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:34:21.642+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:34:21.657+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:34:22.198+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:34:53.780+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:34:54.001+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:34:54.413+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:35:25.582+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:35:26.062+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:35:26.309+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:35:57.265+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:35:57.302+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:35:58.141+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:36:29.377+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:36:30.303+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:36:30.336+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:37:02.141+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:37:02.395+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:37:02.524+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:37:34.360+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:37:34.644+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:37:35.222+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:38:06.877+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:38:07.254+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:38:07.262+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:38:39.066+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:38:39.143+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:38:40.003+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:39:11.108+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:39:11.155+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:39:12.009+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:39:43.307+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:39:44.112+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:39:44.255+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:40:15.435+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:40:15.940+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:40:16.023+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:40:47.876+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:40:48.303+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:40:48.476+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:41:20.222+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:41:20.527+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:41:21.062+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:41:52.651+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:41:52.782+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:41:53.241+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:42:24.780+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:42:25.245+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:42:25.253+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T12:42:56.669+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:42:57.421+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:42:57.579+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:43:28.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T12:43:28.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T12:43:29.285+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:44:00.592+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:44:01.247+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:44:01.493+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:44:33.331+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:44:33.357+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:44:33.575+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:45:05.259+0000 7f6dd0499700 -1 osd.4 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:45:05.431+0000 7f568d57c700 -1 osd.5 155 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:45:06.057+0000 7f89944ae700 -1 osd.6 155 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:45:35.922+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T12:45:36.321+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:45:36.936+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:46:08.451+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:46:08.881+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:46:09.172+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T12:46:40.498+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T12:46:40.791+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:46:41.358+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:47:12.543+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:47:13.148+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T12:47:13.257+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:47:45.039+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T12:47:45.445+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:47:45.474+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:48:17.550+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:48:17.862+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:48:18.142+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T12:48:49.902+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:48:50.013+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:48:50.424+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:49:22.163+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:49:22.763+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:49:23.164+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:49:54.653+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:49:55.005+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:49:55.214+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:50:26.454+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:50:26.546+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:50:26.589+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:50:58.549+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:50:58.799+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:50:59.435+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:51:31.079+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:51:31.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T12:51:31.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T12:52:02.653+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:52:03.050+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:52:03.587+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:52:34.893+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:52:35.356+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:52:35.541+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:53:07.385+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:53:07.640+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:53:07.860+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:53:39.574+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:53:39.905+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:53:40.504+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:54:11.680+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:54:11.757+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:54:12.547+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:54:43.836+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:54:44.544+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:54:44.551+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:55:16.418+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:55:16.573+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:55:16.981+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:55:48.498+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:55:48.853+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:55:49.323+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:56:21.129+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:56:21.357+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:56:21.466+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:56:53.209+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:56:53.639+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:56:54.169+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:57:25.597+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:57:26.121+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T12:57:26.400+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:57:57.726+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:57:58.291+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:57:58.756+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:58:30.080+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:58:30.688+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:58:30.861+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:59:02.459+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:59:03.042+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:59:03.075+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T12:59:34.297+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:59:34.875+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T12:59:35.166+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:00:06.205+0000 7f6dd0499700 -1 osd.4 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:00:06.917+0000 7f89944ae700 -1 osd.6 156 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:00:07.103+0000 7f568d57c700 -1 osd.5 156 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:00:38.097+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:00:38.422+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:00:38.455+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T13:01:10.183+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:01:10.433+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:01:11.022+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:01:42.205+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:01:42.432+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:01:42.903+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:02:14.639+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:02:14.782+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:02:15.385+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T13:02:46.999+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:02:47.400+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:02:47.934+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:03:19.676+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:03:20.025+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:03:20.187+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:03:51.934+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:03:52.353+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:03:52.460+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:04:24.349+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:04:24.451+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:04:24.980+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:04:56.232+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:04:56.557+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T13:04:57.052+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:05:28.345+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:05:29.058+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:05:29.110+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:06:01.023+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:06:01.296+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:06:01.533+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:06:33.593+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:06:33.720+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:06:34.007+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:07:05.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T13:07:05.748+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:07:05.794+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:07:37.739+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:07:37.973+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:07:38.193+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:08:09.934+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:08:10.150+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:08:10.907+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:08:42.055+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:08:42.061+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:08:42.988+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:09:14.129+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:09:15.015+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:09:15.071+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:09:47.032+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:09:47.134+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:09:47.136+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:10:18.986+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:10:19.137+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:10:19.229+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:10:51.201+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:10:51.297+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:10:51.983+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:11:23.176+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:11:23.816+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:11:24.144+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:11:55.818+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:11:56.065+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:11:56.470+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:12:28.049+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:12:28.137+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:12:28.754+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:13:00.365+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:13:00.606+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:13:01.088+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:13:32.516+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:13:33.114+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:13:33.332+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:14:05.330+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:14:05.545+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:14:05.578+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:14:37.518 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T13:14:37.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T13:14:37.521+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:15:09.253+0000 7f89944ae700 -1 osd.6 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:15:09.305+0000 7f568d57c700 -1 osd.5 157 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:15:09.644+0000 7f6dd0499700 -1 osd.4 157 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:15:40.227+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:15:40.890+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:15:41.153+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:16:12.472+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:16:12.729+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:16:13.225+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:16:44.883+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:16:45.199+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:16:45.385+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:17:17.236+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:17:17.396+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:17:17.879+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T13:17:49.620+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:17:50.059+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:17:50.435+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:18:21.995+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T13:18:22.308+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:18:22.722+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:18:54.044+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:18:54.855+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:18:54.987+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:19:26.810+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:19:26.946+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:19:27.153+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:19:58.655+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:19:58.861+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:19:59.439+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:20:30.484+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:20:30.705+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:20:31.384+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:21:02.697+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:21:03.385+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:21:03.427+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:21:35.080+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:21:35.362+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:21:35.580+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:22:07.461+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:22:07.657+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:22:08.045+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:22:39.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T13:22:39.777+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:22:39.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T13:23:11.879+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:23:11.910+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:23:12.305+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:23:43.702+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:23:44.029+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:23:44.534+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:24:16.099+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:24:16.674+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:24:16.943+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:24:48.171+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:24:49.018+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:24:49.028+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:25:21.062+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:25:21.097+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:25:21.099+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:25:53.028+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:25:53.119+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:25:53.980+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:26:25.326+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:26:25.900+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:26:26.264+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:26:57.701+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:26:57.842+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T13:26:58.132+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:27:29.893+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:27:30.086+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:27:30.805+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:28:02.036+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:28:02.836+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:28:03.042+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:28:34.739+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:28:35.008+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:28:35.054+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:29:06.961+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:29:07.201+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:29:07.820+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:29:39.066+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:29:39.984+0000 7f568d57c700 -1 osd.5 158 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:29:40.005+0000 7f89944ae700 -1 osd.6 158 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:30:11.102+0000 7f6dd0499700 -1 osd.4 158 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:30:11.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T13:30:11.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T13:30:41.796+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:30:41.905+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:30:42.017+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:31:13.764+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:31:13.803+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:31:13.908+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:31:45.830+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:31:45.861+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:31:46.672+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:32:18.030+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:32:18.549+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:32:18.603+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:32:50.298+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:32:50.521+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:32:51.074+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:33:22.439+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:33:23.082+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:33:23.402+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:33:55.315+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:33:55.430+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:33:55.567+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:34:27.401+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:34:27.491+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:34:28.300+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T13:34:59.235+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:34:59.311+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:35:00.116+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:35:31.589+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:35:32.315+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:35:32.344+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:36:04.333+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:36:04.399+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:36:04.724+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:36:36.374+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:36:36.982+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:36:37.210+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:37:09.068+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:37:09.330+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:37:09.366+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:37:41.283+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:37:41.422+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:37:41.751+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:38:13.341+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:38:13.538+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T13:38:13.750 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T13:38:44.644+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:38:45.228+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:38:45.298+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:39:17.531+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:39:17.565+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:39:17.877+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:39:49.552+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:39:49.992+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:39:50.430+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:40:21.522+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:40:22.206+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:40:22.257+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:40:54.062+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:40:54.245+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:40:54.538+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:41:26.197+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:41:26.537+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:41:27.119+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:41:58.686+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:41:58.774+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T13:41:58.864+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:42:30.646+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:42:30.820+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:42:30.930+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:43:02.628+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:43:03.046+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:43:03.597+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:43:35.298+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:43:35.714+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:43:35.864+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:44:07.736+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:44:07.802+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:44:08.222+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:44:39.973+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:44:40.305+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:44:40.907+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:45:12.539+0000 7f89944ae700 -1 osd.6 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:45:12.760+0000 7f568d57c700 -1 osd.5 159 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:45:12.916+0000 7f6dd0499700 -1 osd.4 159 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:45:43.650+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:45:43.956+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:45:43.990+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T13:46:15.936+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:46:15.941 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T13:46:15.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T13:46:47.919+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:46:47.942+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:46:48.269+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:47:20.052+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:47:20.281+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:47:20.831+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:47:52.097+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:47:52.805+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:47:53.013+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:48:24.923+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:48:24.945+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:48:25.235+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:48:56.871+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:48:57.034+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:48:57.643+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:49:29.090+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:49:29.208+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T13:49:29.792+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:50:00.916+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:50:00.978+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:50:01.701+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:50:33.137+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:50:33.610+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:50:33.684+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:51:05.650+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:51:05.973+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:51:06.056+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:51:37.812+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:51:38.285+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:51:38.688+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:52:10.276+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:52:10.391+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:52:10.850+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:52:42.520+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:52:42.830+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:52:43.248+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:53:14.446+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T13:53:14.915+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:53:15.297+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:53:46.805+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:53:47.150+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:53:47.540+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:54:18.963+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:54:19.167+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:54:19.349+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:54:50.949+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:54:51.572+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:54:51.874+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:55:23.491+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:55:23.646+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:55:24.024+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:55:55.676+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:55:55.999+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:55:56.385+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:56:27.708+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:56:28.407+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:56:28.692+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:56:59.923+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:57:00.411+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:57:00.457+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:57:32.277+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:57:32.433+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:57:32.785+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:58:04.676+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:58:04.679+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:58:05.456+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:58:36.884+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:58:37.391+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:58:37.541+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:59:09.372+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:59:09.534+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:59:10.035+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T13:59:41.519+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:59:41.914+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T13:59:42.386+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:00:13.818+0000 7f6dd0499700 -1 osd.4 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:00:13.819+0000 7f89944ae700 -1 osd.6 160 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:00:14.219+0000 7f568d57c700 -1 osd.5 160 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:00:44.780+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:00:44.897+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:00:45.069+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:01:17.034+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:01:17.225+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:01:17.984+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:01:49.103+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:01:49.906+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:01:49.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T14:02:21.297+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:02:21.359+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:02:22.090+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:02:53.244+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:02:54.214+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:02:54.214+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:03:26.163+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:03:26.251+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:03:26.679+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:03:58.109+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:03:58.689+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:03:59.027+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:04:30.129+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:04:30.761+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:04:30.975+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:05:02.612+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:05:02.956+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:05:03.054+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:05:35.094+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:05:35.113+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:05:35.727+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:06:07.122+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:06:07.356+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:06:08.005+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:06:39.441+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:06:40.124+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:06:40.153+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:07:12.175+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:07:12.399+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:07:12.629+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:07:44.389+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:07:44.569+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:07:45.143+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:08:16.357+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:08:16.699+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:08:17.278+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:08:48.692+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:08:48.846+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:08:49.500+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:09:21.067+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:09:21.627+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:09:21.950+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:09:53.152+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:09:53.216+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:09:53.660+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:10:25.092+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:10:25.215+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:10:25.472+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:10:57.273+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:10:57.335+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:10:58.012+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:11:29.330+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:11:30.103+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:11:30.213+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:12:01.208+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:12:01.906+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:12:02.008+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:12:33.871+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:12:34.254+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:12:34.576+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:13:06.184+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:13:06.332+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:13:06.811+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:13:38.374+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:13:38.789+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:13:39.117+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:14:10.447+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:14:11.168+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:14:11.224+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:14:42.925+0000 7f568d57c700 -1 osd.5 161 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:14:43.228+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:14:43.726+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:15:15.449+0000 7f6dd0499700 -1 osd.4 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:15:15.475+0000 7f89944ae700 -1 osd.6 161 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:15:15.802+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T14:15:46.416+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:15:46.591+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T14:15:47.349+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:16:18.618+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T14:16:19.354+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:16:19.433+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:16:51.274+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:16:51.558+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T14:16:51.561+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:17:23.553+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T14:17:23.769+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:17:24.309+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:17:55.234+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T14:17:55.523+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:17:55.968+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:18:27.695+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:18:27.927+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:18:28.529+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:18:59.922+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:19:00.384+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:19:00.591+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:19:32.182+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:19:32.363+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:19:32.639+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:20:04.165+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:20:04.329+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:20:04.599+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:20:36.892+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:20:36.991+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:20:37.159+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:21:08.948+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:21:09.318+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:21:09.857+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:21:41.476+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:21:41.678+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:21:41.834+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:22:13.641+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:22:13.717+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:22:14.577+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:22:45.980+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:22:46.227+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:22:46.467+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:23:18.270+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:23:18.408+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:23:18.675+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:23:50.394+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:23:50.617+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:23:51.120+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:24:22.758+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:24:22.945+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:24:23.225+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:24:55.368+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:24:55.371+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:24:55.512+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:25:27.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T14:25:27.160+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:25:27.471+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:25:58.826+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:25:59.597+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:25:59.626+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:26:31.016+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:26:31.456+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:26:31.513+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:27:03.070+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:27:03.569+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:27:03.987+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:27:35.627+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:27:35.821+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:27:36.109+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:28:07.916+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:28:08.142+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:28:08.498+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:28:40.394+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:28:40.505+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:28:41.147+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:29:12.380+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:29:12.927+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:29:13.332+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:29:44.827+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:29:45.208+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:29:45.497+0000 7f568d57c700 -1 osd.5 162 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:30:16.683+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:30:16.984+0000 7f89944ae700 -1 osd.6 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:30:17.292+0000 7f6dd0499700 -1 osd.4 162 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:30:48.092+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:30:48.469+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T14:30:48.490+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:31:20.512+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:31:20.579+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T14:31:21.106+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:31:52.779+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T14:31:52.809+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:31:53.524+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:32:24.920+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T14:32:25.498+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:32:25.792+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:32:57.451+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:32:57.639+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:32:57.753+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) Ignoring malparsed backtrace: 2021-08-25T14:33:29.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T14:33:29.435+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:33:29.733+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:34:01.361+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:34:01.450+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:34:01.904+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T14:34:33.433+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:34:33.715+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:34:34.140+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:35:05.652+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:35:06.307+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:35:06.631+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:35:38.349+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:35:38.738+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:35:38.948+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:36:10.880+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:36:10.898+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:36:11.587+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:36:43.442+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:36:43.916+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:36:44.198+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:37:15.629+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:37:16.215+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:37:16.458+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:37:47.938+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:37:48.054+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:37:48.187+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:38:20.096+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:38:20.206+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:38:21.077+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:38:52.398+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:38:52.858+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:38:52.961+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:39:25.079+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:39:25.270+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:39:25.399+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:39:57.229+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:39:57.402+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:39:58.076+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:40:29.445+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:40:29.611+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:40:29.972+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:41:01.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T14:41:01.423+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:41:01.513+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:41:33.082+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:41:33.427+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:41:33.653+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:42:05.589+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:42:05.939+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:42:06.000+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:42:37.909+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:42:38.092+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:42:38.465+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:43:10.180+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:43:10.489+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:43:10.900+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:43:42.279+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:43:42.951+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:43:43.155+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:44:14.860+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:44:15.113+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:44:15.477+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:44:47.153+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:44:47.555+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:44:47.562+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:45:19.308+0000 7f89944ae700 -1 osd.6 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:45:19.350+0000 7f568d57c700 -1 osd.5 163 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:45:19.646+0000 7f6dd0499700 -1 osd.4 163 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:45:50.277+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:45:50.643+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T14:45:51.104+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:46:22.517+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T14:46:23.133+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:46:23.155+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:46:54.835+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:46:55.084+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:46:55.430+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T14:47:27.057+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:47:27.271+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T14:47:27.706+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:47:59.438+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:47:59.598+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:48:00.192+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T14:48:31.497+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:48:32.040+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:48:32.246+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:49:03.598 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T14:49:03.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T14:49:03.735+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:49:35.234+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:49:35.713+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:49:35.840+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:50:07.387+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:50:07.782+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:50:08.018+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:50:39.735+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:50:39.928+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:50:40.180+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:51:12.063+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:51:12.410+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:51:12.990+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:51:44.351+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:51:44.902+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:51:45.310+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:52:16.857+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:52:17.305+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:52:17.542+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:52:48.767+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:52:49.138+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:52:49.740+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:53:21.059+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:53:21.699+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:53:21.918+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:53:53.682+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:53:53.912+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:53:54.194+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:54:26.122+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:54:26.143+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:54:26.945+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:54:58.370+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:54:59.100+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:54:59.119+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:55:31.142+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:55:31.306+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:55:31.402+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:56:03.161+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:56:03.323+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:56:03.339+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:56:35.327+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:56:35.348+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:56:35.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T14:57:06.926+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:57:07.311+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:57:07.428+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:57:39.134+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:57:39.546+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:57:39.801+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:58:11.383+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:58:11.679+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:58:11.903+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:58:43.721+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:58:43.785+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:58:44.678+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:59:15.653+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:59:16.378+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:59:16.417+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T14:59:48.395+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T14:59:48.398+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T14:59:48.433+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:00:19.434+0000 7f89944ae700 -1 osd.6 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:00:20.151+0000 7f6dd0499700 -1 osd.4 164 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:00:20.389+0000 7f568d57c700 -1 osd.5 164 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:00:51.413+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:00:51.550+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:00:52.201+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T15:01:23.578+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:01:24.367+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:01:24.419+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:01:56.212+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:01:56.357+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:01:56.595+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:02:28.466+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:02:28.669+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:02:29.447+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:03:00.765+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:03:01.464+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:03:01.656+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T15:03:33.424+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T15:03:33.612+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:03:33.738+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:04:05.613+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:04:05.641+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:04:05.973+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:04:37.479+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:04:38.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T15:04:38.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T15:05:09.572+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:05:09.597+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:05:10.286+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:05:41.168+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:05:41.634+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:05:41.634+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:06:13.535+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:06:13.750+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:06:14.146+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:06:45.743+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:06:46.440+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:06:46.698+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:07:18.333+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:07:18.544+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T15:07:18.574+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:07:50.460+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:07:50.806+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:07:50.976+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:08:22.748+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:08:23.288+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:08:23.427+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:08:55.442+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:08:55.498+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:08:55.685+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:09:27.646+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:09:27.920+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:09:28.510+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:09:59.999+0000 7f29c7ecd700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; 1 osds down; 1 host (4 osds) down; Reduced data availability: 1 pg inactive, 1 pg down, 7 pgs stale; Degraded data redundancy: 38236/153330 objects degraded (24.937%), 11 pgs degraded, 16 pgs undersized; 125 slow ops, oldest one blocked for 23359 sec, daemons [osd.0,osd.4,osd.5,osd.6] have slow ops. , 2021-08-25T15:10:00.086+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:10:00.355+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:10:31.996+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:10:32.384+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:10:32.545+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:11:03.983+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:11:04.301+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:11:04.701+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:11:36.533+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:11:36.588+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:11:37.186+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:12:08.705+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:12:09.044+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:12:09.574+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:12:40.550+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:12:40.916+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:12:40.933+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:13:12.988+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:13:13.269+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:13:13.558+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:13:45.367+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:13:45.563+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:13:46.032+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:14:17.682+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:14:18.074+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:14:18.164+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:14:49.534+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:14:49.971+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:14:50.256+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:15:21.873+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:15:22.404+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:15:22.684+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:15:53.625+0000 7f6dd0499700 -1 osd.4 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:15:53.869+0000 7f568d57c700 -1 osd.5 165 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:15:54.391+0000 7f89944ae700 -1 osd.6 165 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:16:26.090+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:16:26.417+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:16:26.638+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:16:58.252+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:16:58.471+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:16:59.104+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:17:30.359+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:17:30.894+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:17:31.279+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T15:18:03.011+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:18:03.192+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:18:03.360+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:18:34.814+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:18:35.183+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T15:18:35.739+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:19:06.904+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:19:07.534+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:19:07.927+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:19:39.541+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:19:39.921+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:19:40.116+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:20:11.718+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:20:11.925+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:20:12.103+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:20:42.727+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:20:42.897+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:20:43.233+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:21:14.937+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:21:15.151+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:21:15.909+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:21:47.065+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:21:47.871+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:21:48.036+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:22:19.155+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:22:20.118+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:22:20.125+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:22:51.714+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:22:51.940+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:22:52.177+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:23:23.827+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:23:24.239+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:23:24.762+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:23:56.290+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:23:56.881+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:23:56.969+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:24:28.813+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:24:29.134+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:24:29.258+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:25:01.111+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:25:01.551+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:25:01.602+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:25:33.433+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:25:33.674+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:25:34.132+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:26:05.555+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:26:05.630+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:26:06.372+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:26:37.774+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:26:38.328+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:26:38.599+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:27:10.342+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:27:10.839+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:27:10.866+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:27:42.715+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:27:42.718+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:27:43.338+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:28:14.487+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:28:14.730+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:28:14.912+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:28:46.917+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:28:46.977+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:28:47.748+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:29:19.120+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:29:19.541+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T15:29:19.760+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:29:51.132+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:29:51.762+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:29:52.083+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:30:23.357+0000 7f568d57c700 -1 osd.5 166 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:30:23.787+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:30:24.160+0000 7f89944ae700 -1 osd.6 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:30:55.905+0000 7f6dd0499700 -1 osd.4 166 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:30:56.097+0000 7f3f3b9d5700 -1 MDSIOContextBase: failed with -110, restarting... , -1> 2021-08-25T15:30:56.097+0000 7f3f3b9d5700 -1 MDSIOContextBase: failed with -110, restarting... Ignoring malparsed backtrace: 2021-08-25T15:31:27.268+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:31:27.367+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:31:28.212+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:31:59.353+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:32:00.047+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:32:00.264+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:32:31.706+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:32:32.034+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:32:32.260+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:33:04.193+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:33:04.469+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:33:04.739+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T15:33:36.153+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:33:36.550+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:33:36.722+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:34:08.561+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:34:08.689+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:34:09.346+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:34:40.956+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:34:41.373+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:34:41.572+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:35:13.397+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:35:13.613+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:35:14.307+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:35:45.662+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:35:45.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T15:35:45.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T15:36:17.736+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:36:17.892+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:36:18.429+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:36:49.863+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:36:49.954+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T15:36:50.534+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:37:21.668+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:37:22.462+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:37:22.565+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:37:54.444+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:37:54.500+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:37:54.803+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:38:26.477+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:38:26.939+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:38:27.409+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:38:58.855+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:38:59.519+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:38:59.649+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:39:31.594+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:39:31.694+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:39:31.764+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:40:03.516+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:40:03.631+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:40:03.696+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:40:35.446+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:40:35.522+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:40:35.578+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:41:07.551+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:41:07.734+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:41:08.208+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:41:39.681+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:41:40.189+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:41:40.584+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:42:12.175+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:42:12.520+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:42:12.629+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:42:44.620+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:42:44.695+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:42:45.411+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:43:16.611+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:43:17.402+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:43:17.413+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:43:48.548+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:43:48.604+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:43:49.266+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:44:20.420+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:44:20.442+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:44:21.435+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:44:52.787+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:44:53.202+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:44:53.285+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:45:25.013+0000 7f568d57c700 -1 osd.5 167 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:45:25.166+0000 7f89944ae700 -1 osd.6 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:45:25.918+0000 7f6dd0499700 -1 osd.4 167 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:45:56.349 INFO:tasks.ceph.mds.f.smithi167.stdout:starting mds.f at , 2021-08-25T15:45:56.813+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:45:57.042+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:46:28.838+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:46:29.191+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:46:29.265+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:47:01.044+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:47:01.245+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:47:01.828+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:47:33.120+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:47:33.547+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:47:34.062+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:48:05.498+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:48:05.627+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T15:48:05.975+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:48:37.445+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:48:38.066+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:48:38.314+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:49:10.271+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:49:10.509+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:49:10.527+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:49:42.633+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:49:42.685+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:49:43.612+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:50:14.582+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:50:14.909+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:50:15.512+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:50:46.849+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:50:47.400+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:50:47.525+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:51:19.484+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:51:19.818+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:51:19.990+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:51:50.929+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:51:51.132+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:51:51.646+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:52:23.321+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:52:23.613+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:52:23.964+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:52:55.483+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:52:56.119+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:52:56.188+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:53:27.953+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:53:28.430+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:53:28.485+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:54:00.405+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:54:00.481+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:54:00.767+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:54:32.469+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:54:32.663+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:54:33.410+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:55:04.872+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:55:05.299+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:55:05.554+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:55:36.883+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:55:37.424+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:55:37.721+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:56:09.467+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:56:09.962+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:56:10.018+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:56:41.998+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:56:42.179+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:56:42.342+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:57:13.988+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:57:14.527+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:57:14.962+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:57:46.276+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:57:46.875+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:57:47.033+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:58:19.003+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:58:19.341+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:58:19.341+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:58:51.229+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:58:51.259+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T15:58:51.393+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T15:59:22.127+0000 7f56943e8700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 64145) UID: 0 , 2021-08-25T15:59:22.127+0000 7f6dd7305700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 64145) UID: 0 , 2021-08-25T15:59:22.127+0000 7f899b31a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 64145) UID: 0 Ignoring malparsed backtrace: 2021-08-25T15:59:54.094+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:59:54.351+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T15:59:55.068+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:00:26.955+0000 7f568d57c700 -1 osd.5 168 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:00:27.083+0000 7f89944ae700 -1 osd.6 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:00:27.364+0000 7f6dd0499700 -1 osd.4 168 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:00:58.297+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:00:58.975+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:00:59.250+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:01:31.451+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:01:31.784+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:01:32.375+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:02:03.634+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:02:04.263+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:02:04.553+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:02:36.439+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T16:02:36.574+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:02:36.576+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:03:08.222 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:03:08.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:03:08.394+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:03:39.940+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:03:40.739+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:03:40.900+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:04:12.844+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:04:12.919+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:04:12.933+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:04:44.976+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:04:45.342+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:04:45.628+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:05:17.329+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:05:17.690+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:05:18.165+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:05:49.724+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:05:50.170+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:05:50.399+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:06:21.815+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:06:22.103+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:06:22.189+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:06:53.975+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:06:54.364+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:06:54.916+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:07:26.639+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:07:26.788+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:07:26.868+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:07:58.738+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:07:58.810+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:07:59.536+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:08:31.017+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:08:31.582+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:08:31.638+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:09:03.423+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:09:03.781+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:09:04.130+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:09:35.777+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:09:36.147+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:09:36.614+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:10:07.595+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:10:07.950+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:10:08.071+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:10:39.905+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:10:40.007+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:10:40.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T16:11:11.425+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:11:12.129+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:11:12.291+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:11:44.116+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:11:44.458+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:11:44.472+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:12:16.478+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:12:16.699+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:12:16.823+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:12:48.413+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:12:49.118+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:12:49.372+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:13:21.300+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:13:21.491+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:13:21.577+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:13:53.243+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:13:53.753+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:13:53.776+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:14:25.638+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:14:25.782+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:14:26.197+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:14:57.823+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:14:58.055+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:14:58.298+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:15:30.152+0000 7f568d57c700 -1 osd.5 169 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:15:30.162+0000 7f89944ae700 -1 osd.6 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:15:30.894+0000 7f6dd0499700 -1 osd.4 169 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:16:01.409+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:16:01.773+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T16:16:02.344+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:16:33.814+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T16:16:34.578+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:16:34.754+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:17:06.369+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:17:06.555+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:17:06.724+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) Ignoring malparsed backtrace: 2021-08-25T16:17:38.160+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:17:38.771+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:17:38.832+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) Ignoring malparsed backtrace: 2021-08-25T16:18:10.661+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T16:18:10.855+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:18:11.229 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T16:18:42.501+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:18:42.641+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T16:18:43.101+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:19:14.772+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:19:15.219+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:19:15.654+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:19:47.146+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:19:47.581+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:19:47.585+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:20:19.184+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:20:19.321+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:20:19.619+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:20:51.553+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:20:51.931+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:20:52.000+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:21:23.811+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:21:23.898+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:21:24.165+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:21:55.981+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:21:56.091+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:21:56.932+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:22:28.036+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:22:28.952+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:22:28.995+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:23:00.789+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:23:01.091+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:23:01.150+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:23:33.088+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:23:33.303+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:23:33.772+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:24:05.476+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:24:05.712+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:24:06.119+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:24:37.701+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:24:37.804+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:24:37.916+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:25:09.814+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:25:10.096+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:25:10.748+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:25:42.068+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:25:42.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:25:42.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T16:26:13.995+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:26:14.295+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:26:14.624+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:26:46.418+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:26:46.639+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:26:46.949+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:27:18.621+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:27:18.819+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:27:19.626+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:27:50.830+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:27:51.364+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:27:51.726+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:28:23.042+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:28:23.374+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:28:23.769+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:28:55.456+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:28:56.089+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:28:56.202+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:29:28.102+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:29:28.262+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:29:28.757+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:29:59.999+0000 7f29c7ecd700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; 1 osds down; 1 host (4 osds) down; Reduced data availability: 1 pg inactive, 1 pg down, 7 pgs stale; Degraded data redundancy: 38236/153330 objects degraded (24.937%), 11 pgs degraded, 16 pgs undersized; 129 slow ops, oldest one blocked for 28159 sec, daemons [osd.0,osd.4,osd.5,osd.6] have slow ops. , 2021-08-25T16:30:00.179+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:30:00.728+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:30:32.212+0000 7f89944ae700 -1 osd.6 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:30:32.687+0000 7f568d57c700 -1 osd.5 170 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:30:33.015+0000 7f6dd0499700 -1 osd.4 170 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:31:03.715+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:31:03.993+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:31:04.211+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) Ignoring malparsed backtrace: 2021-08-25T16:31:36.021+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:31:36.240+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T16:31:36.891+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:32:08.043+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:32:08.156+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T16:32:08.927+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:32:40.338+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:32:40.968+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:32:41.113+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) Ignoring malparsed backtrace: 2021-08-25T16:33:13.048+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:33:13.286+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:33:13.334+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) Ignoring malparsed backtrace: 2021-08-25T16:33:44.463+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T16:33:45.120+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:33:45.358+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:34:17.257+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:34:17.378+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:34:17.383+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:34:49.355+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:34:49.565+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:34:50.143+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:35:21.669+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:35:21.897+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:35:22.526+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:35:53.514+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:35:54.045+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:35:54.289+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:36:26.124+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:36:26.327+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:36:26.612+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:36:58.423+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:36:58.597+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:36:58.994+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:37:30.781+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:37:31.002+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:37:31.407+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:38:02.725+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:38:03.559+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:38:03.656+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:38:35.498+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:38:35.624+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:38:35.775+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:39:07.677+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:39:08.018+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:39:08.585+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T16:39:39.796+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:39:40.110+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:39:40.754+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:40:12.002+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:40:12.090+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:40:12.500+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:40:44.252+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:40:44.417+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:40:44.802+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:41:16.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:41:16.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:41:16.826+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:41:48.440+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:41:48.678+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:41:48.882+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:42:20.863+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:42:21.081+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:42:21.169+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:42:52.988+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:42:53.213+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:42:53.821+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T16:43:24.938+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:43:25.058+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:43:25.895+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:43:57.131+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:43:57.682+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:43:57.753+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:44:29.740+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:44:29.763+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:44:30.167+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:45:01.829+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:45:02.257+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:45:02.607+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:45:34.198+0000 7f89944ae700 -1 osd.6 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:45:34.530+0000 7f6dd0499700 -1 osd.4 171 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:45:34.803+0000 7f568d57c700 -1 osd.5 171 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:46:05.521+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T16:46:05.721+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:46:06.439+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:46:37.591+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T16:46:38.422+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:46:38.502+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:47:09.650+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:47:10.379+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:47:10.405+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T16:47:42.476+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:47:42.484+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T16:47:42.576+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:48:14.434+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T16:48:14.522+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:48:15.405+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:48:46.786+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:48:47.241+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:48:47.362+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T16:49:18.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:49:18.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T16:49:18.832+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:49:50.610+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:49:51.046+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:49:51.478+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:50:22.754+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:50:22.902+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:50:23.310+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:50:54.678+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:50:54.929+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:50:55.352+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:51:26.818+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:51:27.104+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:51:27.662+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:51:59.165+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:51:59.870+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:51:59.956+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:52:31.791+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:52:31.889+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:52:32.176+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:53:04.061+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:53:04.364+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:53:04.672+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:53:36.354+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:53:36.560+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:53:37.086+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:54:08.617+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:54:08.903+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:54:09.383+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:54:40.825+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:54:41.062+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:54:41.276+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:55:12.953+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:55:12.964+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:55:13.816+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:55:45.254+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:55:45.819+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:55:46.065+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:56:17.973+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:56:18.026+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:56:18.118+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:56:50.092+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:56:50.512+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:56:50.917+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:57:21.700+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:57:22.099+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:57:22.709+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:57:54.086+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:57:54.721+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T16:57:54.815+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:58:26.113+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:58:26.418+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:58:27.075+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:58:58.365+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:58:58.776+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:58:59.143+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T16:59:30.669+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:59:31.054+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T16:59:31.249+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:00:02.652+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:00:02.861+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:00:03.540+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:00:35.015+0000 7f6dd0499700 -1 osd.4 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:00:35.515+0000 7f568d57c700 -1 osd.5 172 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:00:35.775+0000 7f89944ae700 -1 osd.6 172 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:01:06.575+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:01:07.033+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:01:07.039+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T17:01:39.097+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:01:39.297+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:01:39.466+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:02:10.972+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:02:11.366+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:02:11.457+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:02:42.994+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:02:43.241+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:02:44.024+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T17:03:15.370+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:03:15.719+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:03:16.239+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:03:47.939+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:03:48.188+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:03:48.427+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:04:20.454+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:04:20.630+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:04:20.942+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:04:52.764+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:04:53.189 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T17:04:53.193 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T17:05:24.305+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:05:24.787+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:05:25.152+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T17:05:56.210+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:05:56.567+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:05:57.141+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:06:28.572+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:06:28.765+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:06:29.339+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:07:00.865+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:07:01.199+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:07:01.742+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:07:33.241+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:07:33.607+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:07:33.827+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:08:05.530+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:08:05.587+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:08:06.056+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:08:37.711+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:08:38.067+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:08:38.592+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:09:10.300+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:09:10.387+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T17:09:10.586+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:09:42.342+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:09:42.513+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:09:42.821+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:10:14.217+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:10:14.606+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:10:14.990+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:10:46.720+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:10:46.784+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:10:46.896+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:11:18.877+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:11:19.013+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:11:19.572+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:11:51.010+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:11:51.447+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:11:51.857+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:12:23.397+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:12:24.161+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:12:24.285+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:12:55.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T17:12:55.481+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:12:55.582+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T17:13:27.314+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:13:27.372+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:13:28.074+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:13:59.354+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:14:00.092+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:14:00.269+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:14:32.132+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:14:32.277+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:14:32.608+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:15:04.306+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:15:04.500+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:15:05.280+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:15:36.710+0000 7f6dd0499700 -1 osd.4 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:15:37.221+0000 7f89944ae700 -1 osd.6 173 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:15:37.473+0000 7f568d57c700 -1 osd.5 173 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:16:08.206+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:16:08.464+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:16:08.571+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T17:16:40.519+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:16:40.568+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:16:40.792+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T17:17:12.605+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:17:12.936+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:17:13.366+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:17:44.838+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:17:45.500+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:17:45.577+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:18:17.577+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:18:17.617+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:18:17.633+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:18:49.604+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T17:18:49.756+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:18:50.330+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:19:21.689+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:19:22.143+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:19:22.704+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:19:54.270+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:19:54.735+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:19:54.802+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:20:26.016+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T17:20:26.349+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:20:26.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T17:20:57.863+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:20:58.026+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:20:58.233+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:21:30.201+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:21:30.474+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:21:31.144+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:22:02.660+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:22:03.054+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:22:03.451+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:22:35.151+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:22:35.383+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:22:35.822+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:23:07.510+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:23:07.672+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:23:08.010+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:23:39.652+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:23:40.132+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:23:40.141+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:24:11.840+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:24:12.303+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:24:12.428+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:24:44.376+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:24:44.557+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:24:44.732+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:25:16.486+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:25:16.598+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:25:17.417+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:25:48.593+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:25:49.313+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:25:49.585+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:26:21.432+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:26:21.582+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:26:21.582+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:26:53.604+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:26:53.638+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:26:54.375+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:27:25.800+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:27:26.450+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T17:27:26.451+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:27:57.843+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:27:58.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T17:27:58.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T17:28:29.560+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:28:29.699+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:28:30.426+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:29:02.199+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:29:02.407+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:29:02.513+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:29:34.327+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:29:34.508+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:29:35.354+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:30:06.336+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:30:06.387+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:30:07.259+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:30:38.598+0000 7f89944ae700 -1 osd.6 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:30:39.194+0000 7f568d57c700 -1 osd.5 174 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:30:39.414+0000 7f6dd0499700 -1 osd.4 174 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:31:10.154+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:31:10.361+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:31:10.503+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:31:42.186+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:31:42.367+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:31:42.595+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:32:14.459+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:32:14.835+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:32:15.075+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:32:46.953+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:32:47.019+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:32:47.184+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:33:19.188+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:33:19.218+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:33:19.487+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:33:51.171+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:33:51.560+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:33:52.019+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:34:23.398+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:34:24.067+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:34:24.380+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:34:55.939+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:34:56.415+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:34:56.484+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:35:27.745+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:35:28.268+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:35:28.406+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:36:00.314+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:36:00.391+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:36:00.553 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T17:36:31.864+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:36:32.193+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:36:32.432+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:37:04.220+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:37:04.478+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:37:04.701+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:37:36.378+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:37:36.459+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:37:37.257+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:38:08.469+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:38:09.173+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:38:09.247+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:38:41.220+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:38:41.398+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:38:41.463+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:39:12.969+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:39:13.188+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:39:13.559+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:39:45.245+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:39:45.603+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:39:45.812+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:40:17.197+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:40:17.530+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:40:17.769+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:40:49.442+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:40:49.591+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:40:50.187+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:41:21.693+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:41:22.173+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:41:22.337+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:41:54.249+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:41:54.587+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:41:54.805+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:42:26.382+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:42:26.672+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:42:26.989+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:42:58.128+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:42:58.710+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:42:59.068+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:43:30.651+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:43:31.146+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:43:31.240+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:44:02.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T17:44:02.790+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:44:03.000+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:44:34.849+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:44:35.022+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:44:35.316+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:45:07.136+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:45:07.516+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:45:08.035+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:45:39.309+0000 7f89944ae700 -1 osd.6 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:45:40.036+0000 7f6dd0499700 -1 osd.4 175 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:45:40.308+0000 7f568d57c700 -1 osd.5 175 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:46:11.060+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:46:11.081+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:46:11.458+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:46:43.140+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:46:43.365+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:46:43.752+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:47:15.579+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:47:15.727+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:47:16.190+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:47:47.705+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:47:48.348+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:47:48.354+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:48:20.058+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:48:20.415+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:48:20.553+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:48:52.297+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:48:52.873+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:48:53.162+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:49:24.747+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:49:25.260+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:49:25.264+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:49:57.171+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:49:57.272+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:49:57.809+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T17:50:28.985+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:50:29.313+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:50:29.339+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:51:01.414+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:51:01.583+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:51:02.216+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:51:33.593+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:51:34.212+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:51:34.479+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:52:05.407+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:52:05.643+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:52:06.167+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:52:37.614+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:52:38.058+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:52:38.212+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:53:09.906+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:53:10.058+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:53:10.657+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:53:42.227+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:53:42.865+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:53:43.026+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T17:54:14.084+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:54:14.861+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:54:15.073+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:54:46.830+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:54:46.924+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:54:47.132+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:55:19.051+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:55:19.064+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:55:19.912+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:55:51.712+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:55:51.737+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:55:52.296+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:56:23.890+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:56:24.536+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:56:24.694+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:56:56.296+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:56:56.695+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:56:57.063+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:57:28.563+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:57:28.807+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:57:28.926+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:58:01.005+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:58:01.164+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:58:01.486+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:58:33.329+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:58:33.773+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:58:34.086+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:59:05.738+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:59:05.992+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:59:06.399+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T17:59:37.295+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:59:37.652+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T17:59:37.976+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:00:09.194+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:00:09.621+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:00:09.901+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:00:41.412+0000 7f6dd0499700 -1 osd.4 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:00:41.729+0000 7f568d57c700 -1 osd.5 176 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:00:42.120+0000 7f89944ae700 -1 osd.6 176 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:01:12.799+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:01:12.996+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:01:13.356+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T18:01:44.802+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:01:44.925+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:01:45.700+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:02:17.038+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:02:17.357+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:02:17.822+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:02:49.181+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:02:49.586+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:02:49.954+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:03:21.567+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:03:21.990+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:03:22.067+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:03:53.889+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:03:54.015+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:03:54.537+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:04:26.108+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:04:26.628+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:04:26.855+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:04:58.592+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T18:04:58.724+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:04:58.727+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:05:30.643+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:05:30.665+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:05:31.025+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:06:02.891+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:06:03.009+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:06:03.375+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:06:35.226+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:06:35.604+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:06:35.619+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:07:07.522+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:07:07.703+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:07:08.095+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:07:39.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:07:39.232+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:07:39.803+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:08:11.525+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:08:11.901+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:08:12.283+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:08:43.684+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:08:43.825+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T18:08:44.297+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:09:15.604+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:09:16.162+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:09:16.575+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:09:48.373+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:09:48.543+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:09:48.852+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:10:20.426+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:10:20.496+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:10:20.912+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:10:52.365+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:10:53.089+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:10:53.285+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:11:24.991+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:11:25.186+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:11:25.512+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:11:57.050+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:11:57.571+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:11:57.998+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:12:29.290+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:12:29.617+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:12:30.082+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:13:01.598+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:13:02.268+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:13:02.356+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:13:34.470+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:13:34.495+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:13:34.691+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:14:06.610+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:14:06.617+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:14:07.512+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:14:38.626+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:14:39.245+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:14:39.390+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:15:11.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:15:11.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:15:11.339+0000 7f6dd0499700 -1 osd.4 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:15:42.576+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:15:43.383+0000 7f568d57c700 -1 osd.5 177 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:15:43.569+0000 7f89944ae700 -1 osd.6 177 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:16:14.252+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T18:16:14.411+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T18:16:14.435+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:16:46.284+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T18:16:46.479+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:16:46.576+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:17:18.448+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:17:18.660+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:17:19.067+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) Ignoring malparsed backtrace: 2021-08-25T18:17:50.581+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:17:51.223+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T18:17:51.435+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:18:23.373+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T18:18:23.696+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:18:23.698+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:18:55.513+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.2332:5 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e154) , 2021-08-25T18:18:55.696+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:18:56.472+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:19:27.862+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:19:28.277+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:19:28.661+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:19:59.999+0000 7f29c7ecd700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; 1 osds down; 1 host (4 osds) down; Reduced data availability: 1 pg inactive, 1 pg down, 7 pgs stale; Degraded data redundancy: 38236/153330 objects degraded (24.937%), 11 pgs degraded, 16 pgs undersized; 164 slow ops, oldest one blocked for 34759 sec, daemons [osd.0,osd.4,osd.5,osd.6] have slow ops. , 2021-08-25T18:20:00.260+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:20:00.427+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:20:32.145+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:20:32.466+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:20:32.558+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:21:04.386+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:21:04.613+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:21:04.916+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:21:36.494+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:21:36.588+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:21:37.330+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:22:08.619+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:22:09.406+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:22:09.483+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:22:41.294+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:22:41.534+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:22:41.812+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:23:12.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:23:13.177+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:23:13.385+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:23:44.959+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:23:45.252+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:23:45.497+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:24:17.268+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:24:17.274+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:24:18.045+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:24:49.473+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:24:49.972+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:24:50.142+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:25:22.072+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:25:22.080+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:25:22.396+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:25:54.186+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:25:54.339+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:25:54.968+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:26:26.089+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:26:26.962+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:26:27.005+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:26:58.844+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:26:59.045+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:26:59.128+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:27:30.945+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:27:31.021+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:27:31.254+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:28:02.888+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:28:03.135+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:28:03.864+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:28:34.929+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:28:35.729+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:28:35.841+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:29:07.299+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:29:07.836+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:29:08.222+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:29:39.688+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:29:39.909+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:29:40.475+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:30:11.581+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:30:11.968+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:30:12.415+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:30:43.882+0000 7f6dd0499700 -1 osd.4 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:30:44.254+0000 7f89944ae700 -1 osd.6 178 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:30:44.790+0000 7f568d57c700 -1 osd.5 178 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:31:15.119+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T18:31:15.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:31:15.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T18:31:47.126+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:31:47.241+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T18:31:47.308+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:32:19.283+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T18:32:19.465+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:32:19.949+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:32:51.559+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:32:51.932+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:32:52.190+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) Ignoring malparsed backtrace: 2021-08-25T18:33:24.189+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:33:24.439+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T18:33:24.540+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:33:56.606+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.1464:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e134) , 2021-08-25T18:33:56.703+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:33:57.210+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:34:28.738+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:34:29.059+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:34:29.710+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:35:00.980+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:35:01.250+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:35:01.766+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:35:33.339+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:35:33.878+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:35:34.277+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:36:06.035+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:36:06.456+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:36:06.533+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:36:38.567+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:36:38.713+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:36:39.230+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:37:10.749+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:37:11.434+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:37:11.722+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:37:43.448+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:37:43.560+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:37:43.732+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:38:15.408+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:38:15.428+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:38:15.459+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T18:38:47.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:38:47.382+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:38:47.473+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:39:19.307+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:39:19.379+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:39:19.494+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:39:51.404+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:39:51.533+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:39:52.289+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:40:23.718+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:40:23.740+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:40:24.126+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:40:55.895+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:40:56.408+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:40:56.897+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:41:28.495+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:41:28.962+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:41:29.145+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:42:00.669+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T18:42:00.866+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:42:01.142+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:42:32.802+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:42:33.203+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:42:33.620+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:43:05.146+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:43:05.251+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:43:06.058+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:43:37.256+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:43:37.499+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:43:38.079+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:44:09.482+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:44:09.889+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:44:10.370+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:44:41.662+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:44:42.213+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:44:42.442+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:45:14.333+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:45:14.387+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:45:14.890+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:45:46.330+0000 7f6dd0499700 -1 osd.4 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:45:46.406+0000 7f568d57c700 -1 osd.5 179 get_health_metrics reporting 53 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:45:46.692+0000 7f89944ae700 -1 osd.6 179 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:46:17.463+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:46:17.515+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:46:18.090+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T18:46:49.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T18:46:49.554+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:46:49.675+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:47:21.326+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:47:21.552+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:47:22.009+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T18:47:53.613+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:47:53.849+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T18:47:54.473+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:48:25.966+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) , 2021-08-25T18:48:26.573+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:48:26.738+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:48:58.687+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:48:58.732+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:48:58.898+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.591:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e127) Ignoring malparsed backtrace: 2021-08-25T18:49:30.669+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:49:30.920+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:49:31.116+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T18:50:02.882+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:50:02.909+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:50:02.980+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:50:34.948+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:50:34.953+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:50:35.561+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:51:07.053+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:51:07.416+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:51:07.833+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:51:39.471+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:51:39.873+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:51:40.008+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:52:11.756+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:52:12.113+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:52:12.408+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:52:44.233+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:52:44.391+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:52:44.626+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:53:16.332+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T18:53:16.407+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:53:16.577+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:53:48.191+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:53:48.628+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:53:48.996+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:54:20.793+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:54:21.321+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:54:21.482+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:54:52.772+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:54:52.921+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:54:53.103+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:55:25.046+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:55:25.050+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:55:25.831+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:55:57.092+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:55:58.009+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:55:58.013+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:56:29.662+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:56:30.197+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:56:30.315+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:57:01.965+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:57:02.181+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:57:02.487+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:57:34.058+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:57:34.580+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:57:34.816+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:58:06.466+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:58:06.956+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:58:07.226+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:58:38.995+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:58:39.209+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:58:39.367+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:59:11.217+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:59:11.520+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:59:11.996+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T18:59:43.700+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:59:43.956+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T18:59:44.088+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:00:15.597+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:00:15.898+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:00:16.160+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:00:47.654+0000 7f568d57c700 -1 osd.5 180 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:00:47.752+0000 7f89944ae700 -1 osd.6 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:00:48.169+0000 7f6dd0499700 -1 osd.4 180 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:01:18.783+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:01:19.274+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T19:01:19.732+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:01:51.100+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T19:01:51.706+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:01:51.735+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:02:23.662+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:02:23.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:02:23.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T19:02:55.078+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T19:02:55.557+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:02:55.589+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:03:27.448+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:03:27.548+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:03:27.869+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T19:03:59.711+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:04:00.004+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:04:00.570+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:04:31.930+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:04:32.306+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:04:32.702+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:05:04.478+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:05:04.521+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:05:05.014+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:05:36.474+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:05:37.221+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:05:37.455+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:06:09.114+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:06:09.185+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:06:09.519+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:06:40.913+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:06:41.525+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:06:41.868+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:07:13.628+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:07:13.774+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:07:14.024+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:07:45.735+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:07:46.204+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:07:46.681+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:08:17.997+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:08:18.267+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:08:18.641+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:08:50.349+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:08:50.425+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:08:51.189+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:09:22.495+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:09:23.438+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:09:23.519+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:09:55.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:09:55.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:09:55.554+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:10:26.550+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:10:26.962+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:10:27.253+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:10:59.175+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:10:59.506+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:10:59.682+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:11:31.635+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:11:31.888+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:11:32.224+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:12:03.568+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:12:03.606+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:12:04.431+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:12:35.507+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:12:36.222+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:12:36.384+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:13:08.331+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:13:08.356+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:13:08.445+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:13:40.482+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:13:40.727+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:13:41.478+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:14:12.762+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:14:13.249+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:14:13.518+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:14:45.100+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:14:45.579+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:14:45.975+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:15:17.573+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T19:15:17.779+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:15:17.894+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:15:49.757+0000 7f568d57c700 -1 osd.5 181 get_health_metrics reporting 55 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:15:49.824+0000 7f89944ae700 -1 osd.6 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:15:50.158+0000 7f6dd0499700 -1 osd.4 181 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:16:21.039+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:16:21.312+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T19:16:21.734+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:16:53.271+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T19:16:53.955+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:16:54.157+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:17:25.819+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:17:26.205+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:17:26.285+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T19:17:57.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:17:57.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:17:58.383+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) Ignoring malparsed backtrace: 2021-08-25T19:18:29.570+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:18:30.157+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.297:4 3.6 3.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force e119) , 2021-08-25T19:18:30.263+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:19:02.167+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:19:02.171+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:19:02.641+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:19:34.027+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:19:34.216+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:19:34.768+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:20:06.085+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:20:06.103+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:20:06.752+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:20:38.298+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:20:38.544+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:20:39.136+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:21:10.568+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:21:11.055+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:21:11.410+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:21:43.207+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:21:43.547+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:21:43.703+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:22:15.691+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:22:15.728+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:22:16.157+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:22:47.972+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T19:22:47.993+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:22:48.334+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:23:20.295+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:23:20.526+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:23:20.889+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:23:52.679+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:23:52.861+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:23:53.269+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:24:24.541+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:24:25.237+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:24:25.346+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:24:57.367+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:24:57.486+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:24:57.727+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:25:29.519 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:25:29.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:25:29.579+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:26:01.475+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:26:01.489+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:26:02.050+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:26:32.321+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:26:32.483+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:26:33.207+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied Ignoring malparsed backtrace: 2021-08-25T19:27:05.374+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:27:05.437+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:27:05.511+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:27:38.396+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:27:38.532+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:27:38.662+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:28:11.579+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:28:11.602+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:28:11.728+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:28:44.593+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:28:44.704+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:28:44.980+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:29:16.543+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:29:16.940+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:29:17.439+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:29:48.709+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:29:49.118+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:29:49.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2021-08-25T19:30:19.948+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:30:20.483+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:30:20.891+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:30:52.434+0000 7f6dd0499700 -1 osd.4 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:30:52.945+0000 7f568d57c700 -1 osd.5 182 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:30:53.062+0000 7f89944ae700 -1 osd.6 182 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:31:23.377+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:31:23.924+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:31:24.153+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:31:56.086+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:31:56.100+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:31:56.174+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:32:28.211+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:32:28.235+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:32:28.948+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:33:00.443+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:33:00.915+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:33:01.070+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:33:32.988+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:33:33.152+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:33:33.279+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:34:05.097+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:34:05.139+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:34:05.273+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:34:37.166+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:34:37.266+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:34:37.992+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:35:09.356+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:35:10.081+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:35:10.150+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:35:42.045+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:35:42.265+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:35:42.549+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:36:14.073+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:36:14.644+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:36:14.822+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:36:46.448+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:36:46.960+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:36:47.347+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:37:18.818+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T19:37:19.133+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:37:19.284+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:37:51.181+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:37:51.281+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:37:51.608+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:38:22.485+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:38:23.207+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:38:23.275+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:38:55.034+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:38:55.282+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:38:55.342+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:39:27.325+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:39:27.532+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:39:27.915+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:39:59.558+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:39:59.839+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:39:59.999+0000 7f29c7ecd700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; 1 osds down; 1 host (4 osds) down; Reduced data availability: 1 pg inactive, 1 pg down, 7 pgs stale; Degraded data redundancy: 38236/153330 objects degraded (24.937%), 11 pgs degraded, 16 pgs undersized; 179 slow ops, oldest one blocked for 39559 sec, daemons [osd.0,osd.4,osd.5,osd.6] have slow ops. Ignoring malparsed backtrace: 2021-08-25T19:40:31.361+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:40:31.547+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:40:32.216+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:41:03.653+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:41:04.042+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T19:41:04.070+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:41:35.858+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 49 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:41:36.216+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:41:36.619+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:42:08.326+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:42:08.594+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:42:08.877+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:42:40.656+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:42:40.797+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:42:41.581+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:43:12.833+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:43:13.580+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:43:13.681+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:43:45.471+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:43:45.642+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:43:45.997+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:44:17.725+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:44:18.248+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:44:18.324+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:44:49.685+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:44:50.401+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:44:50.413+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:45:22.370+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:45:22.504+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:45:22.841+0000 7f568d57c700 -1 osd.5 183 get_health_metrics reporting 57 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:45:54.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2021-08-25T19:45:54.467+0000 7f89944ae700 -1 osd.6 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:45:54.599+0000 7f6dd0499700 -1 osd.4 183 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:46:25.418+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:46:25.488+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:46:25.623+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:46:57.577+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:46:57.580+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:46:58.389+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:47:29.559+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:47:30.212+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:47:30.436+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:48:02.179+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:48:02.533+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:48:02.552+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:48:34.466+0000 7fe6cd4a3700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied , 2021-08-25T19:48:34.562+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:48:34.642+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:49:06.554+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:49:06.601+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:49:07.425+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:49:38.712+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:49:39.596+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:49:39.667+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:50:10.724+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:50:11.484+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:50:11.551+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:50:43.381+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:50:43.727+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:50:43.933+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:51:15.702+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:51:15.829+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:51:16.272+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:51:47.902+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:51:48.242+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:51:48.808+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:52:19.904+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:52:20.350+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:52:20.803+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:52:52.337+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:52:52.849+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:52:53.025+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:53:24.930+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:53:25.194+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:53:25.238+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:53:56.294+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:53:56.702+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:53:57.208+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:54:28.767+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:54:29.100+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:54:29.572+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:55:00.822+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:55:01.759+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:55:01.766+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Ignoring malparsed backtrace: 2021-08-25T19:55:33.765+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:55:33.815+0000 7f89944ae700 -1 osd.6 184 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.90:9486 3.0 3.c892bcc0 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) , 2021-08-25T19:55:33.831+0000 7f568d57c700 -1 osd.5 184 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.90:9479 3.4 3.7614352c (undecoded) ondisk+retry+write+known_if_redirected+full_force e108) Found 3 distinct failure reasons Failure: Command failed on smithi100 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 2 jobs: ['6358352', '6358349'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'mds', 'mon', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/alternate-pool}', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Dead: b'2021-08-25T19:56:27.731 INFO:tasks.ceph.osd.4.smithi167.stderr:2021-08-25T19:56:27.728+0000 7f6dd0499700 -1 osd.4 184 get_health_metrics reporting 51 slow ops, oldest is osd_op(mds.0.90:9517 3.6 3.4b2c82a6 (undecoded) ondisk+retry+write+known_if_redirected+full_force e108)' 1 jobs: ['6358353'] suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: "2021-08-25T08:09:03.598417+0000 mon.a (mon.0) 126 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log 1 jobs: ['6358350'] suites: ['clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/workunit/snaps}', 'whitelist_health', 'whitelist_wrongly_marked_down}']