2022-04-02T19:38:09.228 INFO:root:teuthology version: 1.1.0-d877b023
2022-04-02T19:38:09.232 INFO:teuthology.results:subset: '9/32'
2022-04-02T19:38:09.232 INFO:teuthology.results:seed: '3435'
2022-04-03T02:12:26.061 INFO:root:teuthology version: 1.1.0-d877b023
2022-04-03T02:12:26.089 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-04-03T14:12:31.231 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-04-03T14:13:03.027 INFO:scrape:Found 373 jobs
2022-04-03T14:13:09.606 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-11430-gcc9ed509 (cc9ed509562d767c1a5d8fc115593bc1453dfea1) quincy (dev)
,  1: /lib64/libpthread.so.0(+0x12b20) [0x7f89e3f4fb20]
,  2: gsignal()

2022-04-03T14:13:09.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:51:56.243+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:51:56.682+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:51:57.276+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:52:44.560+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:52:45.226+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:52:45.548+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:53:19.701+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:53:20.441+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:53:20.600+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:53:52.774+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:53:53.136+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:53:53.502+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:54:25.862+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:54:26.199+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:54:26.699+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:54:59.095+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:54:59.232+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:54:59.869+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:55:32.140+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:55:32.244+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:55:32.976+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:56:04.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T22:56:04.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T22:56:05.270+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:56:37.651+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:56:37.783+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:56:38.601+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:57:10.546+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:57:10.856+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:57:11.399+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:57:43.595+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:57:43.936+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:57:44.456+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:58:16.610+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:58:16.844+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:58:17.346+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:58:49.595+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:58:49.757+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:58:50.443+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:59:22.882+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:59:23.104+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:59:23.492+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:59:56.034+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T22:59:56.148+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T22:59:56.220+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:00:28.926+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:00:29.011+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:00:29.350+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:01:01.994+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:01:02.069+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:01:02.716+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:01:34.968+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:01:35.255+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:01:35.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:09.620 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-11430-gcc9ed509 (cc9ed509562d767c1a5d8fc115593bc1453dfea1) quincy (dev)
,  1: /lib64/libpthread.so.0(+0x12b20) [0x7f5a13f3eb20]
,  2: [0x5652cfa76010]

2022-04-03T14:13:09.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:03:01.428+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:03:01.826+0000 7f231ad20700 -1 osd.5 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:03:01.907+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:03:34.578+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:03:34.791+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:03:34.938+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:04:06.564+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:04:06.915+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:04:07.426+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:04:39.755+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:04:39.783+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:04:40.618+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:05:12.528+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:05:12.671+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:05:13.303+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.624 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:05:45.600+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:05:45.922+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:05:46.388+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.625 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:06:18.329+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:06:19.095+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:06:19.184+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:06:43.448+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:06:43.912+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:06:44.254+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.627 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:07:04.151+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:07:04.170+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:07:04.174+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:07:23.655+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:07:24.050+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:07:24.096+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.629 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:07:43.184+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:07:43.731+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:07:43.920+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.629 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:08:03.680+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:08:03.881+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:08:03.918+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:08:23.245+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:08:23.672+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:08:23.792+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:08:43.101+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:08:43.121+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:08:43.671+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:09:03.201+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:09:03.551+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:09:03.846+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.632 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:09:23.178+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:09:23.271+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:09:23.501+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:09:42.984+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:09:43.185+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:09:43.345+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:10:03.167+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:10:03.367+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:10:03.570+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:10:22.738+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:10:23.345+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:10:23.361+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.635 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:10:42.590+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:10:42.977+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:10:43.369+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:11:02.814+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:11:03.268+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:11:03.342+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:11:22.589+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:11:22.654+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:11:23.315+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.637 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:11:42.597+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:11:42.640+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:11:42.646+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.638 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:02.597+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:12:02.665+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:12:03.329+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.638 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:22.340+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:12:22.486+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:12:22.489+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.639 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:42.374+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:12:42.403+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:12:42.508+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.640 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:01.213+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:13:01.343+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:13:01.373+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:20.591+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:13:20.827+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:13:21.121+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:40.190+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:13:40.518+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:13:40.556+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:59.917+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:14:00.189+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:14:00.298+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:14:19.628+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:14:19.813+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:14:19.850+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:14:38.945+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:14:39.448+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:14:39.689+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:14:58.865+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:14:59.043+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:14:59.425+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.645 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:18.560+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:15:18.732+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:15:18.744+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.645 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:38.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:15:38.532+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:15:38.548+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:57.978+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:15:57.984+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:15:58.500+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:17.565+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:16:17.781+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:16:17.795+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:37.651+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:16:37.715+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:16:37.919+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:57.124+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:16:57.576+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:16:57.577+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:17:16.545+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:17:16.651+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:17:16.913+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.650 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:17:36.502+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:17:36.624+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:17:36.934+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:17:55.847+0000 7f231ad20700 -1 osd.5 44 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4219.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:17:56.129+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:17:56.482+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:18:15.605+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:18:15.691+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:18:15.778+0000 7fe007a1a700 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:18:36.754+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:18:36.833+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:18:37.417+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:19:08.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:19:08.910+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:19:08.911+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:19:40.826+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:19:40.887+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:19:41.559+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:20:12.990+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:20:13.097+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:20:13.587+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.655 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:20:45.364+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:20:45.690+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:20:45.987+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:21:17.815+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:21:17.988+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:21:18.372+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:21:45.302+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:21:46.018+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:21:46.122+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:22:09.994+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:22:10.054+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:22:10.469+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.658 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:22:34.938+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:22:34.972+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:22:35.097+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.658 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:22:59.063+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:22:59.094+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:22:59.631+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:23:23.411+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:23:23.785+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:23:23.942+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:23:47.848+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:23:47.979+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:23:48.367+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:24:12.306+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:24:12.452+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:24:13.001+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:09.661 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:24:37.135+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:24:37.274+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:24:37.428+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:01.304+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:25:01.356+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:25:01.854+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.663 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:25.945+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:25:25.954+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:25:26.057+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.663 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:50.023+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:25:50.098+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:25:50.645+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:26:14.099+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:26:14.706+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:26:14.766+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.665 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:26:39.240+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:26:39.668+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:26:39.778+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.665 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:03.750+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:27:03.808+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:27:03.947+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.666 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:27.947+0000 7fe007a1a700 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:27:28.009+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:27:28.259+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.667 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:52.215+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:27:52.603+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:27:52.912+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.667 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:28:16.587+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:28:17.061+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:28:17.096+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.668 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:28:41.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:28:41.194+0000 7fe007a1a700 -1 osd.7 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:28:41.409+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:05.824+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:29:05.918+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:29:06.104+0000 7fe007a1a700 -1 osd.7 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:29.969+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:29:30.153+0000 7fe007a1a700 -1 osd.7 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:29:30.717+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.670 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:54.416+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:29:54.893+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:29:54.900+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.671 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:30:18.861+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:30:18.880+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:30:19.215+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:30:43.018+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:30:43.392+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:30:43.616+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:31:08.003+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:31:08.627+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:31:08.726+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:31:32.636+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:31:32.667+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:31:32.800+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:31:56.892+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:31:57.054+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:31:57.687+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:32:21.072+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:32:21.546+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:32:21.833+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:32:45.340+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:32:45.839+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:32:45.993+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.676 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:33:10.508+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:33:10.706+0000 7fe007a1a700 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:33:10.960+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:33:33.863+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:33:34.519+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:33:34.796+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:34:06.654+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:34:06.776+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:34:06.809+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:34:38.772+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:34:38.782+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:34:39.647+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.679 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:35:10.993+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:35:11.594+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:35:11.909+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.679 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:35:42.853+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:35:43.030+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:35:43.201+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:36:15.006+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:36:15.244+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:36:15.617+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.682 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:36:43.359+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:36:43.394+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:36:43.995+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:37:08.306+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:37:08.360+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:37:08.927+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:37:32.276+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:37:32.926+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:37:33.051+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:37:56.846+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:37:56.929+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:37:57.221+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.685 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:38:21.079+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:38:21.253+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:38:21.755+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.685 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:38:45.199+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:38:45.799+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:38:45.849+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:10.400+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:39:10.596+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:39:11.021+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:34.851+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:39:35.094+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:39:35.429+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:59.277+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:39:59.288+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:39:59.671+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:40:23.666+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:40:23.908+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:40:24.248+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:40:47.800+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:40:48.128+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:40:48.353+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:41:12.795+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:41:12.804+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:41:13.046+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:09.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:41:36.996+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:41:37.286+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:41:37.599+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:01.160+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:42:01.504+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:42:01.926+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.692 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:26.037+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:42:26.106+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:42:26.137+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:50.046+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:42:50.177+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:42:50.939+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:43:14.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:43:14.675+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:43:14.855+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.694 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:43:39.266+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:43:39.940+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:43:40.003+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.695 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:03.910+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:44:04.025+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:44:04.095+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:28.062+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:44:28.309+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:44:28.354+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:52.371+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:44:52.551+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:44:53.033+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.697 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:16.552+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:45:16.921+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:45:17.020+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.698 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:41.442+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:45:41.761+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:45:42.101+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.699 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:05.968+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:46:06.120+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:46:06.320+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.699 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:30.141+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:46:30.306+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:46:30.786+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.700 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:54.652+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:46:54.917+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:46:55.023+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:18.877+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:47:19.017+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:47:19.291+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:43.826+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:47:43.935+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:47:43.999+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.702 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:08.009+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:48:08.062+0000 7fe007a1a700 -1 osd.7 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:48:08.913+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.703 WARNING:scrape:Ignoring malparsed backtrace:     -1> 2022-04-02T23:48:31.771+0000 7f16d3466700 -1 MDSIOContextBase: failed with -110, restarting...
, 2022-04-02T23:48:31.772+0000 7f16d4468700 -1 mds.0.purge_queue operator(): Error -110 loading Journaler
, 2022-04-02T23:48:31.810 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at

2022-04-03T14:13:09.703 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:55.738+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:48:56.070+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-02T23:48:56.314+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.704 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:20.141+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-02T23:49:20.258+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:49:20.573+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.705 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:44.862+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:49:44.889+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-02T23:49:45.036+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.706 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:09.008+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:50:09.579+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:50:09.805+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.706 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:33.841+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-02T23:50:33.990+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:50:34.026+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:49.721+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-02T23:50:49.971+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:50:50.006+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:50:50.006453+0000)

2022-04-03T14:13:09.708 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:02.216+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:51:02.258+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:51:02.510+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:02.511357+0000)

2022-04-03T14:13:09.708 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:14.700+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-02T23:51:15.010+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:15.011210+0000)
, 2022-04-02T23:51:15.067+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:15.067804+0000)

2022-04-03T14:13:09.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:27.145+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:51:27.218+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:51:27.323+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.710 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:39.540+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:39.541803+0000)
, 2022-04-02T23:51:39.848+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:51:40.014+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:40.015759+0000)

2022-04-03T14:13:09.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:52.017+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:52.018006+0000)
, 2022-04-02T23:51:52.074+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:52.075648+0000)
, 2022-04-02T23:51:52.149+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:04.428+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:52:04.531+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:04.531581+0000)
, 2022-04-02T23:52:04.549+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:04.550214+0000)

2022-04-03T14:13:09.712 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:16.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:52:17.020+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:17.022259+0000)
, 2022-04-02T23:52:17.062+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:29.112+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:52:29.490+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:52:29.538+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:29.539692+0000)

2022-04-03T14:13:09.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:42.026+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:42.026934+0000)
, 2022-04-02T23:52:42.085+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:42.086354+0000)
, 2022-04-02T23:52:42.126+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.714 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:54.121+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:52:54.303+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:52:54.547+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:54.547775+0000)

2022-04-03T14:13:09.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:06.569+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:06.570311+0000)
, 2022-04-02T23:53:06.613+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:53:07.030+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:07.031566+0000)

2022-04-03T14:13:09.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:19.033+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:19.033746+0000)
, 2022-04-02T23:53:19.092+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:19.094283+0000)
, 2022-04-02T23:53:19.103+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.716 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:31.330+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:53:31.558+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:31.558788+0000)
, 2022-04-02T23:53:31.578+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:31.578648+0000)

2022-04-03T14:13:09.717 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:44.037+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:44.038254+0000)
, 2022-04-02T23:53:44.037+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:53:44.084+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.718 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:56.243+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:53:56.525+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:53:56.565+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:56.567059+0000)

2022-04-03T14:13:09.718 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:08.730+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:54:09.042+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:09.042752+0000)
, 2022-04-02T23:54:09.105+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:09.105490+0000)

2022-04-03T14:13:09.719 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:21.106+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:21.108010+0000)
, 2022-04-02T23:54:21.155+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:54:21.166+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.720 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:33.577+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:33.579210+0000)
, 2022-04-02T23:54:33.597+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:33.598426+0000)
, 2022-04-02T23:54:33.624+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.720 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:46.049+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:46.049549+0000)
, 2022-04-02T23:54:46.113+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:46.113519+0000)
, 2022-04-02T23:54:46.419+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.721 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:58.395+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:54:58.436+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:54:58.586+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:58.587367+0000)

2022-04-03T14:13:09.722 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:10.930+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:55:11.053+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:11.053994+0000)
, 2022-04-02T23:55:11.118+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:11.118822+0000)

2022-04-03T14:13:09.723 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:23.120+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:23.121445+0000)
, 2022-04-02T23:55:23.580+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:55:23.594+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:23.596179+0000)

2022-04-03T14:13:09.723 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:35.750+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:55:35.766+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:55:35.885+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.724 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:47.862+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:55:48.060+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:48.060992+0000)
, 2022-04-02T23:55:48.125+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:48.126882+0000)

2022-04-03T14:13:09.725 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:00.625+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:00.626297+0000)
, 2022-04-02T23:56:00.631+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:56:00.689+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.725 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:13.065+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:13.066177+0000)
, 2022-04-02T23:56:13.093+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:56:13.132+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:13.132298+0000)

2022-04-03T14:13:09.726 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:25.509+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:56:25.614+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:25.615865+0000)
, 2022-04-02T23:56:25.625+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.727 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:37.741+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:56:37.748+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:56:38.070+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:38.071608+0000)

2022-04-03T14:13:09.727 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:50.073+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:50.073880+0000)
, 2022-04-02T23:56:50.139+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:50.140272+0000)
, 2022-04-02T23:56:50.373+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.728 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:02.644+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:02.645557+0000)
, 2022-04-02T23:57:02.683+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:57:03.006+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.729 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:15.145+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:15.145854+0000)
, 2022-04-02T23:57:15.159+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:57:15.418+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.730 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:27.580+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-02T23:57:27.635+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:27.636230+0000)
, 2022-04-02T23:57:27.653+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:27.653466+0000)

2022-04-03T14:13:09.730 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:40.095+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:57:40.151+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:40.151263+0000)
, 2022-04-02T23:57:40.175+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.731 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:52.278+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:57:52.437+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:57:52.448+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.732 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:04.665+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:04.665867+0000)
, 2022-04-02T23:58:05.087+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:05.088566+0000)
, 2022-04-02T23:58:05.150+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.732 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:17.299+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:58:17.507+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:58:17.652+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:17.652742+0000)

2022-04-03T14:13:09.733 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:29.655+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:29.656525+0000)
, 2022-04-02T23:58:29.673+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:29.674500+0000)
, 2022-04-02T23:58:29.677+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.734 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:42.183+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:58:42.281+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:58:42.659+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:42.660729+0000)

2022-04-03T14:13:09.734 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:54.590+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:58:54.664+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:54.664902+0000)
, 2022-04-02T23:58:54.681+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:54.681699+0000)

2022-04-03T14:13:09.735 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:07.101+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:07.102420+0000)
, 2022-04-02T23:59:07.168+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:07.169645+0000)
, 2022-04-02T23:59:07.204+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.736 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:19.171+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:19.172372+0000)
, 2022-04-02T23:59:19.248+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:59:19.672+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:19.673170+0000)

2022-04-03T14:13:09.737 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:31.693+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:31.694028+0000)
, 2022-04-02T23:59:32.107+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:32.107816+0000)
, 2022-04-02T23:59:32.158+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.737 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:44.176+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:44.177899+0000)
, 2022-04-02T23:59:44.231+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-02T23:59:44.680+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:44.681494+0000)

2022-04-03T14:13:09.738 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:56.700+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:56.701952+0000)
, 2022-04-02T23:59:56.745+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:59:57.112+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:57.113124+0000)

2022-04-03T14:13:09.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:09.213+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:00:09.252+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:00:09.256+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:21.329+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:00:21.693+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:21.694042+0000)
, 2022-04-03T00:00:21.709+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:21.709502+0000)

2022-04-03T14:13:09.740 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:34.120+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:34.121055+0000)
, 2022-04-03T00:00:34.155+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:00:34.187+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:34.188460+0000)

2022-04-03T14:13:09.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:46.394+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:00:46.673+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:00:46.701+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:46.702212+0000)

2022-04-03T14:13:09.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:58.721+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:58.721841+0000)
, 2022-04-03T00:00:58.727+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:00:59.126+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:59.126587+0000)

2022-04-03T14:13:09.742 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:11.195+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:11.196565+0000)
, 2022-04-03T00:01:11.407+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:01:11.567+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:23.705+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:01:23.713+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:23.715184+0000)
, 2022-04-03T00:01:23.729+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:23.729899+0000)

2022-04-03T14:13:09.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:36.072+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:01:36.134+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:36.134598+0000)
, 2022-04-03T00:01:36.201+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:36.201927+0000)

2022-04-03T14:13:09.744 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:48.722+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:48.723501+0000)
, 2022-04-03T00:01:48.737+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:01:48.736+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:48.738060+0000)

2022-04-03T14:13:09.745 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:00.781+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:02:00.928+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:02:01.098+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.746 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:13.209+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:13.210125+0000)
, 2022-04-03T00:02:13.613+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:02:13.731+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:13.731831+0000)

2022-04-03T14:13:09.746 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:25.735+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:25.735948+0000)
, 2022-04-03T00:02:25.749+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:25.750197+0000)
, 2022-04-03T00:02:25.859+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.747 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:38.147+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:38.148635+0000)
, 2022-04-03T00:02:38.214+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:38.215473+0000)
, 2022-04-03T00:02:38.398+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:50.391+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:02:50.705+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:02:50.709+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:02.801+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:03:02.825+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:03:03.154+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:03.154299+0000)

2022-04-03T14:13:09.749 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:15.222+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:15.223557+0000)
, 2022-04-03T00:03:15.630+0000 7fe007a1a700 -1 osd.7 59 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:03:15.635+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:27.755+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:27.756454+0000)
, 2022-04-03T00:03:27.769+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:27.770078+0000)
, 2022-04-03T00:03:27.770+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:39.760+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:39.760471+0000)
, 2022-04-03T00:03:39.768+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:03:39.772+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.751 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:51.776+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:51.777628+0000)
, 2022-04-03T00:03:51.876+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:03:51.912+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.752 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:04.233+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:04.234376+0000)
, 2022-04-03T00:04:04.641+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:04:04.685+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:16.772+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:16.772383+0000)
, 2022-04-03T00:04:16.786+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:16.786424+0000)
, 2022-04-03T00:04:16.942+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:28.866+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:04:29.172+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:29.173224+0000)
, 2022-04-03T00:04:29.238+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:29.239731+0000)

2022-04-03T14:13:09.754 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:41.793+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:41.794761+0000)
, 2022-04-03T00:04:41.802+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:04:41.903+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:53.899+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:04:54.087+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:04:54.178+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:54.178774+0000)

2022-04-03T14:13:09.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:06.633+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:05:06.753+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:05:06.789+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:06.789365+0000)

2022-04-03T14:13:09.756 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:18.894+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:05:19.100+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:05:19.183+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:19.184286+0000)

2022-04-03T14:13:09.757 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:31.186+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:31.187173+0000)
, 2022-04-03T00:05:31.252+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:31.253281+0000)
, 2022-04-03T00:05:31.645+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.757 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:43.741+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:05:43.801+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:43.801888+0000)
, 2022-04-03T00:05:43.815+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:43.815915+0000)

2022-04-03T14:13:09.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:55.965+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:05:56.190+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:56.192164+0000)
, 2022-04-03T00:05:56.257+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:56.258684+0000)

2022-04-03T14:13:09.759 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:08.720+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:06:08.765+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:06:08.810+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:08.810688+0000)

2022-04-03T14:13:09.759 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:20.814+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:20.814783+0000)
, 2022-04-03T00:06:20.827+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:20.828252+0000)
, 2022-04-03T00:06:21.075+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.760 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:33.266+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:33.266984+0000)
, 2022-04-03T00:06:33.495+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:06:33.706+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.761 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:45.822+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:45.823380+0000)
, 2022-04-03T00:06:45.835+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:45.836436+0000)
, 2022-04-03T00:06:45.933+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.762 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:56.826+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:56.827405+0000)
, 2022-04-03T00:06:56.839+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:56.840037+0000)
, 2022-04-03T00:06:56.925+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.762 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:07.923+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:07:08.059+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:07:08.207+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:08.207189+0000)

2022-04-03T14:13:09.763 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:19.130+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:07:19.208+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:19.209698+0000)
, 2022-04-03T00:07:19.275+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:19.276753+0000)

2022-04-03T14:13:09.764 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:29.850+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:29.851136+0000)
, 2022-04-03T00:07:30.085+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:07:30.211+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:30.211929+0000)

2022-04-03T14:13:09.764 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:41.213+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:41.214208+0000)
, 2022-04-03T00:07:41.281+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:41.281398+0000)
, 2022-04-03T00:07:41.338+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.765 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:51.992+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:07:52.215+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:52.216607+0000)
, 2022-04-03T00:07:52.282+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:52.283726+0000)

2022-04-03T14:13:09.766 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:03.218+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:03.218943+0000)
, 2022-04-03T00:08:03.285+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:03.286197+0000)
, 2022-04-03T00:08:03.549+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.766 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:14.287+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:14.288559+0000)
, 2022-04-03T00:08:14.430+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:08:14.569+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.767 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:25.290+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:25.291054+0000)
, 2022-04-03T00:08:25.351+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:08:25.543+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.768 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:36.484+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:08:36.636+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:08:36.691+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:47.613+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:08:47.651+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:08:47.723+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:47.724945+0000)

2022-04-03T14:13:09.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:58.297+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:58.298392+0000)
, 2022-04-03T00:08:58.619+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:08:58.692+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.770 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:09.707+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:09:09.731+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:09.731849+0000)
, 2022-04-03T00:09:09.872+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:09.872361+0000)

2022-04-03T14:13:09.771 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:20.813+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:09:20.875+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:20.876023+0000)
, 2022-04-03T00:09:20.887+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:20.888418+0000)

2022-04-03T14:13:09.771 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:31.689+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:09:31.737+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:31.739230+0000)
, 2022-04-03T00:09:31.878+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:31.879463+0000)

2022-04-03T14:13:09.772 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:42.882+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:42.883093+0000)
, 2022-04-03T00:09:42.895+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:42.895834+0000)
, 2022-04-03T00:09:43.149+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.773 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:53.745+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:53.746922+0000)
, 2022-04-03T00:09:53.885+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:53.886609+0000)
, 2022-04-03T00:09:53.898+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:53.899280+0000)

2022-04-03T14:13:09.773 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:04.841+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:10:04.889+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:04.890124+0000)
, 2022-04-03T00:10:04.901+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:04.902840+0000)

2022-04-03T14:13:09.774 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:15.893+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:15.893950+0000)
, 2022-04-03T00:10:15.905+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:15.906446+0000)
, 2022-04-03T00:10:16.247+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:16.248141+0000)

2022-04-03T14:13:09.775 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:26.909+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:26.909974+0000)
, 2022-04-03T00:10:26.962+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:10:27.250+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:27.250684+0000)

2022-04-03T14:13:09.776 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:37.912+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:37.913613+0000)
, 2022-04-03T00:10:38.252+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:38.253132+0000)
, 2022-04-03T00:10:38.318+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:38.320232+0000)

2022-04-03T14:13:09.776 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:49.254+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:49.255586+0000)
, 2022-04-03T00:10:49.321+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:49.322506+0000)
, 2022-04-03T00:10:49.727+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.777 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:59.968+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:11:00.257+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:00.257810+0000)
, 2022-04-03T00:11:00.324+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:00.324854+0000)

2022-04-03T14:13:09.778 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:11.326+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:11.327002+0000)
, 2022-04-03T00:11:11.761+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:11.771+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:11.771930+0000)

2022-04-03T14:13:09.778 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:22.261+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:22.262506+0000)
, 2022-04-03T00:11:22.328+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:22.329462+0000)
, 2022-04-03T00:11:22.774+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:22.775320+0000)

2022-04-03T14:13:09.779 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:33.331+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:33.331898+0000)
, 2022-04-03T00:11:33.778+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:33.779030+0000)
, 2022-04-03T00:11:33.894+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.780 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:44.603+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:44.782+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:44.782829+0000)
, 2022-04-03T00:11:44.923+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:44.924011+0000)

2022-04-03T14:13:09.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:55.550+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:55.785+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:55.786577+0000)
, 2022-04-03T00:11:55.926+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:55.928060+0000)

2022-04-03T14:13:09.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:06.788+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:06.790184+0000)
, 2022-04-03T00:12:06.931+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:06.932021+0000)
, 2022-04-03T00:12:06.943+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:06.944392+0000)

2022-04-03T14:13:09.782 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:17.935+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:17.936084+0000)
, 2022-04-03T00:12:17.947+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:17.948066+0000)
, 2022-04-03T00:12:18.017+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.783 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:28.829+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:12:28.939+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:28.939722+0000)
, 2022-04-03T00:12:28.951+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:28.951675+0000)

2022-04-03T14:13:09.783 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:39.954+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:39.955434+0000)
, 2022-04-03T00:12:40.060+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:12:40.278+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:40.279669+0000)

2022-04-03T14:13:09.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:50.958+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:50.959065+0000)
, 2022-04-03T00:12:51.281+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:51.281893+0000)
, 2022-04-03T00:12:51.347+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:51.348314+0000)

2022-04-03T14:13:09.785 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:01.950+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:01.950713+0000)
, 2022-04-03T00:13:01.962+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:01.962790+0000)
, 2022-04-03T00:13:02.283+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:02.284439+0000)

2022-04-03T14:13:09.785 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:13.180+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T00:13:13.286+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:13.286837+0000)
, 2022-04-03T00:13:13.326+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:23.968+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:23.969781+0000)
, 2022-04-03T00:13:24.288+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:24.289300+0000)
, 2022-04-03T00:13:24.322+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.787 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:35.291+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:35.291862+0000)
, 2022-04-03T00:13:35.357+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:35.357909+0000)
, 2022-04-03T00:13:35.385+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:46.293+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:46.294151+0000)
, 2022-04-03T00:13:46.359+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:46.360239+0000)
, 2022-04-03T00:13:46.478+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:57.362+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:57.362637+0000)
, 2022-04-03T00:13:57.397+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:13:57.407+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.789 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:08.364+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:08.364967+0000)
, 2022-04-03T00:14:08.417+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:14:08.828+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:08.829310+0000)

2022-04-03T14:13:09.790 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:19.483+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:14:19.832+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:19.832997+0000)
, 2022-04-03T00:14:19.896+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.790 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:30.369+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:30.369579+0000)
, 2022-04-03T00:14:30.413+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:14:30.775+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:41.746+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:14:41.838+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:41.840246+0000)
, 2022-04-03T00:14:41.984+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:41.985391+0000)

2022-04-03T14:13:09.792 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:52.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:14:52.697+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:14:52.843+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:52.844359+0000)

2022-04-03T14:13:09.792 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:03.691+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:15:03.798+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:03.847+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:03.848013+0000)

2022-04-03T14:13:09.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:14.841+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:14.847+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:14.851+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:14.851806+0000)

2022-04-03T14:13:09.794 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:25.853+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:25.854+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:25.855322+0000)
, 2022-04-03T00:15:25.861+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:36.905+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:36.979+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:37.003+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:37.003904+0000)

2022-04-03T14:13:09.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:47.981+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:48.007+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:48.007733+0000)
, 2022-04-03T00:15:48.018+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:48.018869+0000)

2022-04-03T14:13:09.796 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:58.955+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:59.011+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:59.011466+0000)
, 2022-04-03T00:15:59.012+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:10.022+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:10.026+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:10.027089+0000)
, 2022-04-03T00:16:10.310+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:21.030+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:21.031013+0000)
, 2022-04-03T00:16:21.203+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:16:21.328+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:21.328590+0000)

2022-04-03T14:13:09.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:32.034+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:32.034781+0000)
, 2022-04-03T00:16:32.111+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:32.328+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:43.259+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:43.326+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:16:43.332+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:43.333320+0000)

2022-04-03T14:13:09.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:54.313+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:54.334+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:54.335710+0000)
, 2022-04-03T00:16:54.400+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:54.400906+0000)

2022-04-03T14:13:09.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:05.337+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:05.338125+0000)
, 2022-04-03T00:17:05.402+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:05.403312+0000)
, 2022-04-03T00:17:05.501+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:16.405+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:16.405735+0000)
, 2022-04-03T00:17:16.438+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:17:16.892+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:16.892569+0000)

2022-04-03T14:13:09.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:27.407+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:27.408143+0000)
, 2022-04-03T00:17:27.489+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:17:27.880+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.802 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:38.519+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:17:38.817+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:17:38.898+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:38.900091+0000)

2022-04-03T14:13:09.803 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:49.885+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:17:49.903+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:49.903866+0000)
, 2022-04-03T00:17:49.976+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:00.565+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:18:00.864+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:18:00.877+0000 7fe007a1a700 -1 osd.7 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:11.909+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:11.910836+0000)
, 2022-04-03T00:18:11.963+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:18:12.055+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:12.056803+0000)

2022-04-03T14:13:09.805 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:23.027+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:18:23.060+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:23.060804+0000)
, 2022-04-03T00:18:23.073+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:23.073861+0000)

2022-04-03T14:13:09.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:33.208+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:18:33.250+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:18:33.356+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:33.357250+0000)

2022-04-03T14:13:09.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:44.274+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:18:44.358+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:44.359676+0000)
, 2022-04-03T00:18:44.424+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:44.425206+0000)

2022-04-03T14:13:09.807 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:55.321+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:18:55.361+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:55.362132+0000)
, 2022-04-03T00:18:55.426+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:55.427520+0000)

2022-04-03T14:13:09.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:06.429+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:06.429819+0000)
, 2022-04-03T00:19:06.538+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:19:06.795+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:17.529+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:19:17.712+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:19:17.829+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:28.433+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:28.434450+0000)
, 2022-04-03T00:19:28.570+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:19:28.661+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.810 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:39.607+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:19:39.939+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:39.940679+0000)
, 2022-04-03T00:19:39.962+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.810 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:50.943+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:50.943919+0000)
, 2022-04-03T00:19:51.089+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:51.089416+0000)
, 2022-04-03T00:19:51.104+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:51.104776+0000)

2022-04-03T14:13:09.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:01.629+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:20:01.947+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:01.947559+0000)
, 2022-04-03T00:20:01.993+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.812 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:12.965+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:20:13.081+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:20:13.095+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:13.096843+0000)

2022-04-03T14:13:09.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:24.072+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:20:24.099+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:24.100454+0000)
, 2022-04-03T00:20:24.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:09.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:34.957+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:34.958297+0000)
, 2022-04-03T00:20:35.067+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:20:35.104+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:35.104471+0000)

2022-04-03T14:13:09.814 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:46.051+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:20:46.107+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:46.108422+0000)
, 2022-04-03T00:20:46.122+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:46.123415+0000)

2022-04-03T14:13:09.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:57.111+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:57.112422+0000)
, 2022-04-03T00:20:57.124+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:20:57.125+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:57.127256+0000)

2022-04-03T14:13:09.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:08.114+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:08.116070+0000)
, 2022-04-03T00:21:08.130+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:08.130810+0000)
, 2022-04-03T00:21:08.384+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.816 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:19.219+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:21:19.364+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:21:19.393+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:19.393463+0000)

2022-04-03T14:13:09.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:30.137+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:30.138794+0000)
, 2022-04-03T00:21:30.159+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:21:30.299+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:41.288+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:21:41.397+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:41.398163+0000)
, 2022-04-03T00:21:41.462+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:41.463257+0000)

2022-04-03T14:13:09.818 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:52.400+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:52.400585+0000)
, 2022-04-03T00:21:52.465+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:52.465745+0000)
, 2022-04-03T00:21:52.565+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.819 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:03.402+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:03.402904+0000)
, 2022-04-03T00:22:03.428+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:22:03.468+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:03.468302+0000)

2022-04-03T14:13:09.820 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:14.469+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:14.470675+0000)
, 2022-04-03T00:22:14.532+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:22:14.662+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.820 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:25.472+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:25.473059+0000)
, 2022-04-03T00:22:25.557+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:22:25.679+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.821 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:36.540+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:22:36.766+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:22:36.995+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.822 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:47.684+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:22:48.001+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:48.002856+0000)
, 2022-04-03T00:22:48.002+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.822 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:58.598+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:22:58.715+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:22:58.816+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:09.936+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:23:10.009+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:10.010319+0000)
, 2022-04-03T00:23:10.156+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:10.157193+0000)

2022-04-03T14:13:09.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:21.013+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:21.014200+0000)
, 2022-04-03T00:23:21.160+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:21.161006+0000)
, 2022-04-03T00:23:21.177+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:21.178179+0000)

2022-04-03T14:13:09.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:31.882+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:23:32.017+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:32.018067+0000)
, 2022-04-03T00:23:32.163+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:32.164645+0000)

2022-04-03T14:13:09.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:43.167+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:43.168468+0000)
, 2022-04-03T00:23:43.185+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:43.186425+0000)
, 2022-04-03T00:23:43.229+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:09.826 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:54.189+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:54.190225+0000)
, 2022-04-03T00:23:54.426+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:54.427261+0000)
, 2022-04-03T00:23:54.435+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.826 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:05.176+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:05.176212+0000)
, 2022-04-03T00:24:05.193+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:05.194117+0000)
, 2022-04-03T00:24:05.429+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:05.429814+0000)

2022-04-03T14:13:09.827 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:16.197+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:16.198190+0000)
, 2022-04-03T00:24:16.421+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:24:16.431+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:16.432318+0000)

2022-04-03T14:13:09.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:27.200+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:27.201981+0000)
, 2022-04-03T00:24:27.353+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:24:27.397+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:38.324+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:24:38.430+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:24:38.436+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:38.436412+0000)

2022-04-03T14:13:09.829 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:49.437+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:49.438532+0000)
, 2022-04-03T00:24:49.449+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:24:49.461+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:00.316+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:25:00.392+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:25:00.397+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.831 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:11.444+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:25:11.448+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:25:11.507+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:25:11.508101+0000)

2022-04-03T14:13:09.831 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:22.546+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:25:22.616+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:25:22.757+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.832 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:33.459+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:25:33.511+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:25:33.512743+0000)
, 2022-04-03T00:25:33.626+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:44.514+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:25:44.515171+0000)
, 2022-04-03T00:25:44.587+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:25:44.621+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:55.693+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:25:55.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:25:55.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:09.834 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:06.557+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:26:06.743+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:26:07.070+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:07.070569+0000)

2022-04-03T14:13:09.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:17.792+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:26:18.060+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:26:18.073+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:18.074102+0000)

2022-04-03T14:13:09.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:28.587+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:26:28.949+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:26:29.077+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:29.078029+0000)

2022-04-03T14:13:09.836 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:40.081+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:40.081551+0000)
, 2022-04-03T00:26:40.202+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:26:40.229+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:40.229601+0000)

2022-04-03T14:13:09.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:51.233+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:51.233361+0000)
, 2022-04-03T00:26:51.233+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:26:51.251+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:51.251734+0000)

2022-04-03T14:13:09.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:02.088+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:02.089132+0000)
, 2022-04-03T00:27:02.228+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:02.235+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:02.236870+0000)

2022-04-03T14:13:09.838 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:13.244+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T00:27:13.251+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:13.259+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:13.259746+0000)

2022-04-03T14:13:09.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:24.315+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:24.423+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:24.471+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:24.471635+0000)

2022-04-03T14:13:09.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:35.247+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:35.248616+0000)
, 2022-04-03T00:27:35.266+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:35.267950+0000)
, 2022-04-03T00:27:35.472+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:35.474008+0000)

2022-04-03T14:13:09.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:46.271+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:46.272090+0000)
, 2022-04-03T00:27:46.381+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:46.475+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:46.476237+0000)

2022-04-03T14:13:09.841 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:57.275+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:57.276062+0000)
, 2022-04-03T00:27:57.399+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:57.478+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:57.478599+0000)

2022-04-03T14:13:09.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:08.314+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:28:08.480+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:08.480915+0000)
, 2022-04-03T00:28:08.545+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:08.545424+0000)

2022-04-03T14:13:09.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:19.482+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:19.483288+0000)
, 2022-04-03T00:28:19.546+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:19.547823+0000)
, 2022-04-03T00:28:19.892+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.843 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:30.292+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:28:30.484+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:30.485450+0000)
, 2022-04-03T00:28:30.549+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:30.550265+0000)

2022-04-03T14:13:09.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:41.552+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:41.552430+0000)
, 2022-04-03T00:28:42.076+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:28:42.118+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:52.985+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:28:53.127+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:53.127895+0000)
, 2022-04-03T00:28:53.153+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:03.555+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:03.556993+0000)
, 2022-04-03T00:29:04.015+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:29:04.118+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:15.018+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:29:15.064+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:29:15.090+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:26.110+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:29:26.135+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:29:26.138+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:26.139013+0000)

2022-04-03T14:13:09.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:37.122+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:29:37.140+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:29:37.141+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:37.142624+0000)

2022-04-03T14:13:09.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:48.145+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:48.146437+0000)
, 2022-04-03T00:29:48.160+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:29:48.175+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:59.145+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:29:59.149+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:59.150370+0000)
, 2022-04-03T00:29:59.180+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:10.165+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:30:10.195+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:30:10.305+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:10.306008+0000)

2022-04-03T14:13:09.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:21.198+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:30:21.308+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:21.309840+0000)
, 2022-04-03T00:30:21.327+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:21.328175+0000)

2022-04-03T14:13:09.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:32.268+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:30:32.313+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:32.314053+0000)
, 2022-04-03T00:30:32.330+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:32.332057+0000)

2022-04-03T14:13:09.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:43.297+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:30:43.317+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:43.317777+0000)
, 2022-04-03T00:30:43.336+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:43.336367+0000)

2022-04-03T14:13:09.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:54.339+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:54.339978+0000)
, 2022-04-03T00:30:54.516+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:54.516750+0000)
, 2022-04-03T00:30:54.580+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:54.580959+0000)

2022-04-03T14:13:09.853 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:05.343+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:05.343881+0000)
, 2022-04-03T00:31:05.400+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:31:05.447+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.853 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:16.347+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:16.347849+0000)
, 2022-04-03T00:31:16.503+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:31:16.521+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:16.521736+0000)

2022-04-03T14:13:09.854 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:27.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:31:27.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:31:27.587+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:27.587856+0000)

2022-04-03T14:13:09.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:38.525+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:38.526593+0000)
, 2022-04-03T00:31:38.588+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:38.590188+0000)
, 2022-04-03T00:31:38.682+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:49.591+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:49.592480+0000)
, 2022-04-03T00:31:49.786+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:31:49.892+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.856 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:00.530+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:00.531548+0000)
, 2022-04-03T00:32:00.594+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:00.594921+0000)
, 2022-04-03T00:32:00.679+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:11.680+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:32:11.798+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:32:11.883+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:22.739+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:32:23.016+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:32:23.199+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:23.200564+0000)

2022-04-03T14:13:09.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:33.742+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:32:33.744+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:32:33.987+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:44.849+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:32:45.170+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:32:45.208+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:45.208381+0000)

2022-04-03T14:13:09.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:56.211+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:56.212037+0000)
, 2022-04-03T00:32:56.363+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:56.364218+0000)
, 2022-04-03T00:32:56.382+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:56.382722+0000)

2022-04-03T14:13:09.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:07.201+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:33:07.215+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:07.216013+0000)
, 2022-04-03T00:33:07.307+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.861 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:18.211+0000 7fe007a1a700 -1 osd.7 61 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:33:18.218+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:18.219462+0000)
, 2022-04-03T00:33:18.371+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:18.372226+0000)

2022-04-03T14:13:09.862 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:29.125+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:33:29.205+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:33:29.222+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:29.223138+0000)

2022-04-03T14:13:09.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:39.615+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:39.616707+0000)
, 2022-04-03T00:33:40.097+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:33:40.222+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:51.023+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:33:51.230+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:51.231154+0000)
, 2022-04-03T00:33:51.285+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.864 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:01.621+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:01.621579+0000)
, 2022-04-03T00:34:02.142+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:34:02.234+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:02.235458+0000)

2022-04-03T14:13:09.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:13.238+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:13.239282+0000)
, 2022-04-03T00:34:13.271+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T00:34:13.392+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:13.392703+0000)

2022-04-03T14:13:09.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:24.396+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:24.397114+0000)
, 2022-04-03T00:34:24.398+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:34:24.414+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:24.415774+0000)

2022-04-03T14:13:09.866 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:35.246+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:35.247080+0000)
, 2022-04-03T00:34:35.400+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:35.401392+0000)
, 2022-04-03T00:34:35.418+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:35.419564+0000)

2022-04-03T14:13:09.867 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:46.250+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:46.251065+0000)
, 2022-04-03T00:34:46.405+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:46.405710+0000)
, 2022-04-03T00:34:46.422+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:46.423290+0000)

2022-04-03T14:13:09.867 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:57.427+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:57.427634+0000)
, 2022-04-03T00:34:57.568+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:57.568874+0000)
, 2022-04-03T00:34:57.633+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:57.633419+0000)

2022-04-03T14:13:09.868 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:08.412+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:08.413808+0000)
, 2022-04-03T00:35:08.430+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:08.431669+0000)
, 2022-04-03T00:35:08.570+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:08.571138+0000)

2022-04-03T14:13:09.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:19.435+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:19.436125+0000)
, 2022-04-03T00:35:19.573+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:19.573655+0000)
, 2022-04-03T00:35:19.636+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:19.638108+0000)

2022-04-03T14:13:09.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:30.439+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:30.440198+0000)
, 2022-04-03T00:35:30.474+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:35:30.575+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:30.575820+0000)

2022-04-03T14:13:09.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:41.576+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:41.578149+0000)
, 2022-04-03T00:35:41.642+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:41.643133+0000)
, 2022-04-03T00:35:41.889+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:52.644+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:52.645620+0000)
, 2022-04-03T00:35:52.889+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:35:52.919+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:03.602+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:36:03.647+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:03.647843+0000)
, 2022-04-03T00:36:03.753+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.872 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:14.783+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:36:14.859+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:36:14.939+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.873 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:25.913+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:36:26.246+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:36:26.285+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:26.286625+0000)

2022-04-03T14:13:09.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:36.833+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:36:36.979+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:36:37.196+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:48.101+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:36:48.276+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:36:48.294+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:48.294370+0000)

2022-04-03T14:13:09.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:59.135+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:36:59.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:36:59.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:09.876 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:10.232+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:37:10.301+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:10.302370+0000)
, 2022-04-03T00:37:10.457+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:10.457945+0000)

2022-04-03T14:13:09.876 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:21.336+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:37:21.461+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:21.462387+0000)
, 2022-04-03T00:37:21.479+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:21.480907+0000)

2022-04-03T14:13:09.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:32.313+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:37:32.323+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:37:32.466+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:32.466888+0000)

2022-04-03T14:13:09.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:43.312+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:43.313476+0000)
, 2022-04-03T00:37:43.469+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:43.470916+0000)
, 2022-04-03T00:37:43.488+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:43.489144+0000)

2022-04-03T14:13:09.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:54.492+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:54.493050+0000)
, 2022-04-03T00:37:54.606+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:54.607668+0000)
, 2022-04-03T00:37:54.670+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:54.671444+0000)

2022-04-03T14:13:09.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:05.478+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:05.479548+0000)
, 2022-04-03T00:38:05.496+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:05.497341+0000)
, 2022-04-03T00:38:05.609+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:05.609910+0000)

2022-04-03T14:13:09.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:16.501+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:16.501366+0000)
, 2022-04-03T00:38:16.612+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:16.612400+0000)
, 2022-04-03T00:38:16.675+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:16.676122+0000)

2022-04-03T14:13:09.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:27.677+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:27.678485+0000)
, 2022-04-03T00:38:27.776+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:38:27.959+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.881 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:38.616+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:38.617230+0000)
, 2022-04-03T00:38:38.679+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:38.680732+0000)
, 2022-04-03T00:38:38.866+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:49.682+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:49.683293+0000)
, 2022-04-03T00:38:49.977+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:38:50.086+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:00.621+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:00.621919+0000)
, 2022-04-03T00:39:00.685+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:00.685703+0000)
, 2022-04-03T00:39:00.927+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:11.834+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:39:12.069+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:39:12.249+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.884 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:23.175+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:39:23.345+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:39:23.347+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:23.347702+0000)

2022-04-03T14:13:09.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:33.921+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:39:34.116+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:39:34.195+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:45.214+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:39:45.354+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:45.355667+0000)
, 2022-04-03T00:39:45.358+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.886 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:56.416+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:39:56.436+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:39:56.519+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:56.520106+0000)

2022-04-03T14:13:09.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:07.362+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:07.363757+0000)
, 2022-04-03T00:40:07.390+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:40:07.393+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:18.367+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:18.367858+0000)
, 2022-04-03T00:40:18.402+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:40:18.528+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:18.528510+0000)

2022-04-03T14:13:09.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:29.531+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:29.532568+0000)
, 2022-04-03T00:40:29.548+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:29.549504+0000)
, 2022-04-03T00:40:29.603+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.889 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:40.494+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:40:40.535+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:40.536473+0000)
, 2022-04-03T00:40:40.552+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:40.553454+0000)

2022-04-03T14:13:09.890 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:51.539+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:51.540769+0000)
, 2022-04-03T00:40:51.556+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:51.557156+0000)
, 2022-04-03T00:40:51.645+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:51.647248+0000)

2022-04-03T14:13:09.890 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:02.387+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:41:02.544+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:02.545067+0000)
, 2022-04-03T00:41:02.560+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:02.561116+0000)

2022-04-03T14:13:09.891 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:13.548+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:13.549275+0000)
, 2022-04-03T00:41:13.564+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:13.565128+0000)
, 2022-04-03T00:41:13.650+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:13.651866+0000)

2022-04-03T14:13:09.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:24.653+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:24.654365+0000)
, 2022-04-03T00:41:24.716+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:24.717344+0000)
, 2022-04-03T00:41:24.957+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:35.573+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:35.573915+0000)
, 2022-04-03T00:41:35.656+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:35.656885+0000)
, 2022-04-03T00:41:35.719+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:35.719623+0000)

2022-04-03T14:13:09.893 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:46.658+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:46.659159+0000)
, 2022-04-03T00:41:46.721+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:46.721924+0000)
, 2022-04-03T00:41:47.040+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.894 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:58.074+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:41:58.108+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:41:58.119+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.894 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:08.726+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:08.726585+0000)
, 2022-04-03T00:42:09.032+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:42:09.211+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.895 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:20.057+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:42:20.262+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:42:20.280+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.896 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:30.730+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:30.731408+0000)
, 2022-04-03T00:42:30.951+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:42:31.196+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:42.301+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:42:42.419+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:42.420300+0000)
, 2022-04-03T00:42:42.448+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:53.410+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:42:53.424+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:53.424661+0000)
, 2022-04-03T00:42:53.558+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.899 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:04.160+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:43:04.428+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:04.428884+0000)
, 2022-04-03T00:43:04.539+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.900 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:15.431+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:15.432730+0000)
, 2022-04-03T00:43:15.435+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:43:15.593+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:15.594377+0000)

2022-04-03T14:13:09.900 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:26.597+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:26.598654+0000)
, 2022-04-03T00:43:26.614+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:26.614778+0000)
, 2022-04-03T00:43:26.633+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.901 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:37.440+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:37.441043+0000)
, 2022-04-03T00:43:37.602+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:37.602689+0000)
, 2022-04-03T00:43:37.603+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.902 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:48.606+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:48.607048+0000)
, 2022-04-03T00:43:48.622+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:48.622927+0000)
, 2022-04-03T00:43:48.651+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.903 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:59.610+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:59.611409+0000)
, 2022-04-03T00:43:59.625+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:59.626733+0000)
, 2022-04-03T00:43:59.688+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:59.689061+0000)

2022-04-03T14:13:09.903 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:10.614+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:10.615390+0000)
, 2022-04-03T00:44:10.629+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:10.630485+0000)
, 2022-04-03T00:44:10.690+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:10.691561+0000)

2022-04-03T14:13:09.904 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:21.634+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:21.634397+0000)
, 2022-04-03T00:44:21.693+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:21.694072+0000)
, 2022-04-03T00:44:21.755+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:21.755498+0000)

2022-04-03T14:13:09.905 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:32.622+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:32.623881+0000)
, 2022-04-03T00:44:32.637+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:32.638705+0000)
, 2022-04-03T00:44:32.756+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:32.757628+0000)

2022-04-03T14:13:09.905 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:43.642+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:43.643129+0000)
, 2022-04-03T00:44:43.697+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:43.698711+0000)
, 2022-04-03T00:44:43.759+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:43.759786+0000)

2022-04-03T14:13:09.906 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:54.761+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:54.762152+0000)
, 2022-04-03T00:44:54.960+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:44:55.096+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.907 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:05.703+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:05.704026+0000)
, 2022-04-03T00:45:05.764+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:05.764452+0000)
, 2022-04-03T00:45:06.000+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.907 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:16.765+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:16.766820+0000)
, 2022-04-03T00:45:16.924+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:45:17.073+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.908 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:28.044+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:45:28.238+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:45:28.309+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:09.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:38.906+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:45:39.082+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:45:39.222+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:50.126+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:45:50.222+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:45:50.426+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.910 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:01.317+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:01.380+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:01.492+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:01.494226+0000)

2022-04-03T14:13:09.911 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:12.320+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:12.357+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:12.497+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:12.498281+0000)

2022-04-03T14:13:09.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:23.316+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:23.501+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:23.502050+0000)
, 2022-04-03T00:46:23.588+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:34.287+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:34.319+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:34.506+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:34.506253+0000)

2022-04-03T14:13:09.913 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:45.415+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:45.509+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:45.510555+0000)
, 2022-04-03T00:46:45.672+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:45.673165+0000)

2022-04-03T14:13:09.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:56.676+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:56.677288+0000)
, 2022-04-03T00:46:56.692+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:56.693015+0000)
, 2022-04-03T00:46:56.713+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:07.518+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:07.518512+0000)
, 2022-04-03T00:47:07.596+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:47:07.681+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:07.681543+0000)

2022-04-03T14:13:09.915 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:18.583+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:47:18.684+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:18.685601+0000)
, 2022-04-03T00:47:18.700+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:18.701274+0000)

2022-04-03T14:13:09.916 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:29.735+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:29.736680+0000)
, 2022-04-03T00:47:29.795+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:29.795827+0000)
, 2022-04-03T00:47:29.796+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.916 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:40.738+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:40.739050+0000)
, 2022-04-03T00:47:40.777+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:47:40.794+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.917 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:51.716+0000 7fe007a1a700 -1 osd.7 62 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:47:51.741+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:51.741356+0000)
, 2022-04-03T00:47:51.800+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:51.800316+0000)

2022-04-03T14:13:09.918 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:02.742+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:02.743703+0000)
, 2022-04-03T00:48:02.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:48:02.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:09.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:13.720+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:13.722166+0000)
, 2022-04-03T00:48:13.745+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:13.745996+0000)
, 2022-04-03T00:48:13.804+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:13.805055+0000)

2022-04-03T14:13:09.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:24.807+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:24.807444+0000)
, 2022-04-03T00:48:25.236+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:48:25.509+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.920 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:35.596+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:48:35.605+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:48:35.692+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:46.662+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:48:46.718+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:46.720235+0000)
, 2022-04-03T00:48:46.734+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:46.735182+0000)

2022-04-03T14:13:09.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:57.738+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:57.739058+0000)
, 2022-04-03T00:48:57.754+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:57.755095+0000)
, 2022-04-03T00:48:57.814+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:57.814837+0000)

2022-04-03T14:13:09.922 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:08.728+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:08.728602+0000)
, 2022-04-03T00:49:08.743+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:08.743396+0000)
, 2022-04-03T00:49:08.755+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:19.747+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:19.747907+0000)
, 2022-04-03T00:49:19.759+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:19.759840+0000)
, 2022-04-03T00:49:19.770+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:30.750+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:30.752114+0000)
, 2022-04-03T00:49:30.760+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:30.762134+0000)
, 2022-04-03T00:49:30.820+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:30.821830+0000)

2022-04-03T14:13:09.924 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:41.755+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:41.756115+0000)
, 2022-04-03T00:49:41.763+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:41.764423+0000)
, 2022-04-03T00:49:41.823+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:41.824262+0000)

2022-04-03T14:13:09.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:52.766+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:52.766666+0000)
, 2022-04-03T00:49:52.826+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:52.826687+0000)
, 2022-04-03T00:49:53.250+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:03.763+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:03.764752+0000)
, 2022-04-03T00:50:03.768+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:03.769214+0000)
, 2022-04-03T00:50:03.827+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:03.828809+0000)

2022-04-03T14:13:09.926 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:14.770+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:14.771893+0000)
, 2022-04-03T00:50:14.830+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:14.831188+0000)
, 2022-04-03T00:50:15.383+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.927 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:26.342+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:50:26.590+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:26.592219+0000)
, 2022-04-03T00:50:26.645+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:36.835+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:36.835673+0000)
, 2022-04-03T00:50:37.478+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:50:37.596+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:37.596371+0000)

2022-04-03T14:13:09.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:48.505+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:50:48.591+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:50:48.599+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:48.600261+0000)

2022-04-03T14:13:09.929 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:59.528+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:50:59.538+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:50:59.603+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:59.603804+0000)

2022-04-03T14:13:09.930 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:10.546+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:51:10.594+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:51:10.607+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:10.607804+0000)

2022-04-03T14:13:09.930 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:21.598+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:51:21.611+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:21.611745+0000)
, 2022-04-03T00:51:21.778+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:21.778622+0000)

2022-04-03T14:13:09.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:32.698+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T00:51:32.781+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:32.782820+0000)
, 2022-04-03T00:51:32.788+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:32.789707+0000)

2022-04-03T14:13:09.932 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:43.618+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:43.619274+0000)
, 2022-04-03T00:51:43.718+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:51:43.786+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:43.787103+0000)

2022-04-03T14:13:09.932 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:54.794+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:54.794862+0000)
, 2022-04-03T00:51:54.807+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:54.807457+0000)
, 2022-04-03T00:51:54.818+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.933 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:05.794+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:05.795405+0000)
, 2022-04-03T00:52:05.795+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:05.797192+0000)
, 2022-04-03T00:52:05.810+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:05.811529+0000)

2022-04-03T14:13:09.934 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:16.798+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:16.799518+0000)
, 2022-04-03T00:52:16.813+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:16.815284+0000)
, 2022-04-03T00:52:16.856+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:16.857087+0000)

2022-04-03T14:13:09.935 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:27.858+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:27.859325+0000)
, 2022-04-03T00:52:27.887+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:52:28.282+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.935 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:38.858+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:52:38.861+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:38.861720+0000)
, 2022-04-03T00:52:38.906+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.936 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:49.826+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:49.827715+0000)
, 2022-04-03T00:52:49.862+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:49.863947+0000)
, 2022-04-03T00:52:50.317+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.937 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:00.935+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:53:01.496+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:53:01.646+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:01.648073+0000)

2022-04-03T14:13:09.937 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:11.909+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:53:12.606+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:53:12.651+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:12.652070+0000)

2022-04-03T14:13:09.938 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:23.655+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:23.655959+0000)
, 2022-04-03T00:53:23.682+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:53:23.709+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.939 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:34.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:53:34.659+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:34.660209+0000)
, 2022-04-03T00:53:34.672+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.939 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:45.664+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:45.664636+0000)
, 2022-04-03T00:53:45.752+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:53:45.763+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.940 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:56.802+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:53:56.812+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:53:56.821+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:56.821470+0000)

2022-04-03T14:13:09.941 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:07.673+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:07.673149+0000)
, 2022-04-03T00:54:07.823+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:07.823984+0000)
, 2022-04-03T00:54:07.841+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:07.842017+0000)

2022-04-03T14:13:09.941 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:18.825+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:18.826519+0000)
, 2022-04-03T00:54:18.845+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:18.845908+0000)
, 2022-04-03T00:54:18.859+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.942 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:29.848+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:29.849735+0000)
, 2022-04-03T00:54:29.865+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:29.866093+0000)
, 2022-04-03T00:54:29.882+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:29.884300+0000)

2022-04-03T14:13:09.943 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:40.831+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:40.832001+0000)
, 2022-04-03T00:54:40.853+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:40.854007+0000)
, 2022-04-03T00:54:40.869+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:40.870194+0000)

2022-04-03T14:13:09.944 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:51.856+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:51.857968+0000)
, 2022-04-03T00:54:51.873+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:51.874529+0000)
, 2022-04-03T00:54:51.887+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:51.889124+0000)

2022-04-03T14:13:09.944 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:02.877+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:02.878724+0000)
, 2022-04-03T00:55:02.890+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:02.891667+0000)
, 2022-04-03T00:55:03.117+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.945 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:13.865+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:13.866294+0000)
, 2022-04-03T00:55:13.881+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:13.882841+0000)
, 2022-04-03T00:55:13.892+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:13.894003+0000)

2022-04-03T14:13:09.946 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:24.908+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:55:25.188+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:55:25.197+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.946 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:35.897+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:35.898763+0000)
, 2022-04-03T00:55:36.123+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:55:36.181+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.947 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:47.139+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:55:47.178+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:55:47.432+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.948 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:58.346+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T00:55:58.347+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:55:58.584+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.948 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:09.120+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:56:09.403+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:56:09.462+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.949 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:20.404+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:56:20.527+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:56:20.722+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:20.723687+0000)

2022-04-03T14:13:09.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:31.493+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:56:31.726+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:31.727814+0000)
, 2022-04-03T00:56:31.856+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:31.857445+0000)

2022-04-03T14:13:09.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:42.401+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:56:42.731+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:42.732023+0000)
, 2022-04-03T00:56:42.858+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:53.735+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:53.736029+0000)
, 2022-04-03T00:56:53.861+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:53.862411+0000)
, 2022-04-03T00:56:53.902+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:53.903852+0000)

2022-04-03T14:13:09.952 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:04.863+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:04.864803+0000)
, 2022-04-03T00:57:04.906+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:04.908089+0000)
, 2022-04-03T00:57:04.916+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:04.917650+0000)

2022-04-03T14:13:09.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:15.743+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:15.744108+0000)
, 2022-04-03T00:57:15.866+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:15.867182+0000)
, 2022-04-03T00:57:15.911+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:15.912415+0000)

2022-04-03T14:13:09.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:26.916+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:26.916764+0000)
, 2022-04-03T00:57:26.921+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:26.922267+0000)
, 2022-04-03T00:57:26.933+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:26.933609+0000)

2022-04-03T14:13:09.954 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:37.790+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:57:37.870+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:37.871820+0000)
, 2022-04-03T00:57:37.919+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:37.920877+0000)

2022-04-03T14:13:09.955 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:48.872+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:48.874201+0000)
, 2022-04-03T00:57:48.924+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:48.925077+0000)
, 2022-04-03T00:57:48.925+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:48.927105+0000)

2022-04-03T14:13:09.956 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:59.928+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:59.929214+0000)
, 2022-04-03T00:57:59.928+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:59.929451+0000)
, 2022-04-03T00:57:59.946+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:59.946641+0000)

2022-04-03T14:13:09.956 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:10.931+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:10.931502+0000)
, 2022-04-03T00:58:10.932+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:10.933197+0000)
, 2022-04-03T00:58:10.950+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:10.950894+0000)

2022-04-03T14:13:09.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:21.935+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:21.937249+0000)
, 2022-04-03T00:58:21.954+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:21.955380+0000)
, 2022-04-03T00:58:22.337+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.958 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:32.959+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:32.959663+0000)
, 2022-04-03T00:58:33.156+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:58:33.498+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.958 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:43.945+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:43.945749+0000)
, 2022-04-03T00:58:43.963+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:43.964098+0000)
, 2022-04-03T00:58:44.277+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.959 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:55.354+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:58:55.473+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:58:55.479+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.960 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:05.971+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:05.972511+0000)
, 2022-04-03T00:59:06.245+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:59:06.414+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.960 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:17.331+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:59:17.394+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:59:17.621+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.961 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:28.358+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T00:59:28.594+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:59:28.615+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.962 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:39.380+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T00:59:39.394+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:59:39.474+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.962 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:50.361+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T00:59:50.452+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:59:50.803+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:50.804583+0000)

2022-04-03T14:13:09.963 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:01.545+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:00:01.808+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:01.808719+0000)
, 2022-04-03T01:00:01.901+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:01.902889+0000)

2022-04-03T14:13:09.964 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:12.587+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:00:12.812+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:12.813246+0000)
, 2022-04-03T01:00:12.904+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:12.905297+0000)

2022-04-03T14:13:09.965 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:23.816+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:23.817360+0000)
, 2022-04-03T01:00:23.906+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:23.907495+0000)
, 2022-04-03T01:00:23.959+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:23.960340+0000)

2022-04-03T14:13:09.965 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:34.909+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:34.909844+0000)
, 2022-04-03T01:00:34.961+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:34.962741+0000)
, 2022-04-03T01:00:34.988+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:34.989410+0000)

2022-04-03T14:13:09.966 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:45.825+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:45.825809+0000)
, 2022-04-03T01:00:45.911+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:45.912243+0000)
, 2022-04-03T01:00:45.964+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:45.965071+0000)

2022-04-03T14:13:09.967 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:56.967+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:56.967282+0000)
, 2022-04-03T01:00:56.997+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:56.997615+0000)
, 2022-04-03T01:00:57.016+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:57.017101+0000)

2022-04-03T14:13:09.967 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:07.916+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:07.917085+0000)
, 2022-04-03T01:01:07.968+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:07.969700+0000)
, 2022-04-03T01:01:07.974+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.968 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:18.918+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:18.919521+0000)
, 2022-04-03T01:01:18.971+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:18.971994+0000)
, 2022-04-03T01:01:19.005+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:19.006273+0000)

2022-04-03T14:13:09.969 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:29.974+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:01:30.010+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:30.010577+0000)
, 2022-04-03T01:01:30.029+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:30.029752+0000)

2022-04-03T14:13:09.969 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:41.013+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:41.014678+0000)
, 2022-04-03T01:01:41.020+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:01:41.032+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:41.033788+0000)

2022-04-03T14:13:09.970 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:52.037+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:52.038394+0000)
, 2022-04-03T01:01:52.193+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:01:52.198+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.971 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:03.243+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:02:03.266+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:03.412+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.972 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:14.046+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:02:14.046968+0000)
, 2022-04-03T01:02:14.244+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:02:14.338+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.972 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:25.284+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:25.350+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:25.735+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.973 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:36.320+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:36.650+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:02:36.867+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:02:36.867710+0000)

2022-04-03T14:13:09.974 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:47.241+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:47.352+0000 7fe007a1a700 -1 osd.7 63 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:47.484+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.974 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:58.368+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:02:58.379+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:02:58.876+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:02:58.876382+0000)

2022-04-03T14:13:09.975 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:09.341+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:03:09.376+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:03:09.829+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.976 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:20.464+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:03:20.674+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:03:20.884+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:20.884968+0000)

2022-04-03T14:13:09.976 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:31.783+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:03:31.888+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:31.889012+0000)
, 2022-04-03T01:03:31.947+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:31.948290+0000)

2022-04-03T14:13:09.977 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:42.002+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:42.003285+0000)
, 2022-04-03T01:03:42.062+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:42.062570+0000)
, 2022-04-03T01:03:42.082+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:42.082488+0000)

2022-04-03T14:13:09.978 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:53.066+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:53.066911+0000)
, 2022-04-03T01:03:53.085+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:53.086699+0000)
, 2022-04-03T01:03:53.406+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.978 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:04.141+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:04:04.196+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:04:04.610+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.979 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:15.095+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:15.095451+0000)
, 2022-04-03T01:04:15.159+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:04:15.247+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.980 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:26.364+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:04:26.695+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:04:26.832+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.980 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:37.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:04:37.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:04:37.507+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.981 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:48.465+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:04:48.641+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:04:48.918+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:48.919284+0000)

2022-04-03T14:13:09.982 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:59.486+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:04:59.923+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:59.923662+0000)
, 2022-04-03T01:04:59.967+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.983 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:10.763+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:05:10.910+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:05:10.926+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:10.928157+0000)

2022-04-03T14:13:09.983 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:21.855+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:05:21.910+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:05:21.931+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:21.932535+0000)

2022-04-03T14:13:09.984 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:32.936+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:32.936788+0000)
, 2022-04-03T01:05:32.975+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:32.975503+0000)
, 2022-04-03T01:05:32.983+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.985 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:43.894+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:05:43.940+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:43.941003+0000)
, 2022-04-03T01:05:43.950+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:09.985 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:54.979+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:54.980316+0000)
, 2022-04-03T01:05:55.031+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:55.032602+0000)
, 2022-04-03T01:05:55.078+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.986 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:06.034+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:06.034995+0000)
, 2022-04-03T01:06:06.119+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:06.119978+0000)
, 2022-04-03T01:06:06.138+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:06.139614+0000)

2022-04-03T14:13:09.987 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:17.036+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:17.037183+0000)
, 2022-04-03T01:06:17.051+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:06:17.124+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:17.124419+0000)

2022-04-03T14:13:09.987 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:28.148+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:28.148393+0000)
, 2022-04-03T01:06:28.175+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:06:28.188+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.988 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:39.132+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:39.133180+0000)
, 2022-04-03T01:06:39.152+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:39.152863+0000)
, 2022-04-03T01:06:39.157+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.989 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:50.136+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:50.137420+0000)
, 2022-04-03T01:06:50.156+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:50.157410+0000)
, 2022-04-03T01:06:50.189+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.989 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:01.103+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:07:01.141+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:01.141824+0000)
, 2022-04-03T01:07:01.161+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:01.161775+0000)

2022-04-03T14:13:09.990 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:12.129+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:07:12.145+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:12.146190+0000)
, 2022-04-03T01:07:12.165+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:12.166075+0000)

2022-04-03T14:13:09.991 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:23.149+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:23.150425+0000)
, 2022-04-03T01:07:23.169+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:23.170796+0000)
, 2022-04-03T01:07:23.518+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.992 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:34.174+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:34.175388+0000)
, 2022-04-03T01:07:34.548+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:07:34.864+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:09.992 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:45.159+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:45.159566+0000)
, 2022-04-03T01:07:45.179+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:45.179755+0000)
, 2022-04-03T01:07:45.481+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.993 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:56.477+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:07:56.517+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:07:56.767+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.994 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:07.645+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:08:07.868+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:08:07.970+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.994 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:18.488+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:08:18.701+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:08:18.858+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.995 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:29.650+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:08:29.967+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:08:30.004+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:30.004925+0000)

2022-04-03T14:13:09.996 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:40.817+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:08:40.982+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:08:41.008+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:41.009071+0000)

2022-04-03T14:13:09.996 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:52.003+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:08:52.011+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:08:52.012+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:52.013394+0000)

2022-04-03T14:13:09.997 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:03.022+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:03.022696+0000)
, 2022-04-03T01:09:03.071+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:03.072075+0000)
, 2022-04-03T01:09:03.086+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:09.998 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:14.020+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:14.021445+0000)
, 2022-04-03T01:09:14.023+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:14.025011+0000)
, 2022-04-03T01:09:14.073+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:14.074524+0000)

2022-04-03T14:13:09.998 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:25.075+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:09:25.075+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:25.076912+0000)
, 2022-04-03T01:09:25.158+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:09.999 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:36.029+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:36.029859+0000)
, 2022-04-03T01:09:36.078+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:36.079397+0000)
, 2022-04-03T01:09:36.185+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.000 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:47.031+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:47.032157+0000)
, 2022-04-03T01:09:47.033+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:47.033429+0000)
, 2022-04-03T01:09:47.081+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:47.081674+0000)

2022-04-03T14:13:10.001 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:58.123+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:09:58.209+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:58.210844+0000)
, 2022-04-03T01:09:58.232+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:58.233618+0000)

2022-04-03T14:13:10.001 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:09.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:10:09.172+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:10:09.213+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:09.214766+0000)

2022-04-03T14:13:10.002 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:20.088+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:20.088633+0000)
, 2022-04-03T01:10:20.106+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:10:20.218+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:20.218944+0000)

2022-04-03T14:13:10.003 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:31.222+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:31.222939+0000)
, 2022-04-03T01:10:31.246+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:31.246492+0000)
, 2022-04-03T01:10:31.248+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.003 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:42.226+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:42.227102+0000)
, 2022-04-03T01:10:42.249+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:42.251088+0000)
, 2022-04-03T01:10:42.332+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.004 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:53.254+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:53.255703+0000)
, 2022-04-03T01:10:53.312+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:10:53.666+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.005 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:04.350+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:04.589+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:04.803+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.005 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:15.264+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:15.264636+0000)
, 2022-04-03T01:11:15.306+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:15.568+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.006 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:26.641+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:26.708+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:11:27.003+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.007 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:37.772+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:38.044+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:11:38.056+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:38.056641+0000)

2022-04-03T14:13:10.008 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:48.711+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:11:48.859+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:49.058+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:49.059115+0000)

2022-04-03T14:13:10.008 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:59.827+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:12:00.060+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:00.061439+0000)
, 2022-04-03T01:12:00.084+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:00.085349+0000)

2022-04-03T14:13:10.009 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:10.816+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:12:11.063+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:11.064003+0000)
, 2022-04-03T01:12:11.088+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:11.089711+0000)

2022-04-03T14:13:10.010 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:22.066+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:22.066875+0000)
, 2022-04-03T01:12:22.093+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:22.093994+0000)
, 2022-04-03T01:12:22.114+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:22.114619+0000)

2022-04-03T14:13:10.010 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:33.098+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:33.098352+0000)
, 2022-04-03T01:12:33.116+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:33.117235+0000)
, 2022-04-03T01:12:33.270+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:33.271152+0000)

2022-04-03T14:13:10.011 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:44.071+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:44.072096+0000)
, 2022-04-03T01:12:44.101+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:44.102315+0000)
, 2022-04-03T01:12:44.118+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:44.119613+0000)

2022-04-03T14:13:10.012 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:55.204+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:12:55.279+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:55.280248+0000)
, 2022-04-03T01:12:55.304+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:55.305127+0000)

2022-04-03T14:13:10.012 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:06.284+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:06.284928+0000)
, 2022-04-03T01:13:06.295+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:13:06.309+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:06.309599+0000)

2022-04-03T14:13:10.013 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:17.289+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:17.289782+0000)
, 2022-04-03T01:13:17.313+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:17.313662+0000)
, 2022-04-03T01:13:17.342+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.014 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:28.317+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:28.318381+0000)
, 2022-04-03T01:13:28.339+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:28.401+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.014 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:39.384+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:39.501+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:13:40.009+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.015 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:50.327+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:50.327397+0000)
, 2022-04-03T01:13:50.495+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:13:50.518+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.016 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:01.455+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:14:01.585+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:01.977+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.017 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:12.335+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:12.336813+0000)
, 2022-04-03T01:14:12.610+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:12.821+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.017 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:23.622+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:23.754+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:24.089+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.018 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:34.692+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:35.059+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:14:35.098+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:35.099063+0000)

2022-04-03T14:13:10.019 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:45.631+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:45.721+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:45.934+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.019 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:57.075+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:14:57.103+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:57.104365+0000)
, 2022-04-03T01:14:57.147+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:57.148320+0000)

2022-04-03T14:13:10.020 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:08.150+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:08.150785+0000)
, 2022-04-03T01:15:08.158+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:08.159062+0000)
, 2022-04-03T01:15:08.207+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.021 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:19.109+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:19.109465+0000)
, 2022-04-03T01:15:19.153+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:19.153394+0000)
, 2022-04-03T01:15:19.163+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:19.163426+0000)

2022-04-03T14:13:10.021 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:30.154+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:30.155797+0000)
, 2022-04-03T01:15:30.166+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:30.167482+0000)
, 2022-04-03T01:15:30.259+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.022 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:41.157+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:41.158202+0000)
, 2022-04-03T01:15:41.170+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:41.171885+0000)
, 2022-04-03T01:15:41.266+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.023 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:52.160+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:52.160446+0000)
, 2022-04-03T01:15:52.175+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:52.176261+0000)
, 2022-04-03T01:15:52.352+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:52.353289+0000)

2022-04-03T14:13:10.024 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:03.180+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:03.180541+0000)
, 2022-04-03T01:16:03.357+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:03.357640+0000)
, 2022-04-03T01:16:03.381+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:03.381536+0000)

2022-04-03T14:13:10.024 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:14.164+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:14.165103+0000)
, 2022-04-03T01:16:14.183+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:14.184645+0000)
, 2022-04-03T01:16:14.361+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:14.362397+0000)

2022-04-03T14:13:10.025 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:25.366+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:25.366830+0000)
, 2022-04-03T01:16:25.389+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:25.390455+0000)
, 2022-04-03T01:16:25.451+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.026 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:36.370+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:36.370977+0000)
, 2022-04-03T01:16:36.394+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:36.394746+0000)
, 2022-04-03T01:16:36.516+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.026 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:47.374+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:47.375242+0000)
, 2022-04-03T01:16:47.397+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:47.399141+0000)
, 2022-04-03T01:16:47.467+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.027 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:58.402+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:58.403530+0000)
, 2022-04-03T01:16:58.403+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:16:58.412+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.028 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:09.489+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:17:09.656+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:17:09.766+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.028 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:20.412+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:20.412391+0000)
, 2022-04-03T01:17:20.422+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:17:20.742+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.029 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:31.415+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:31.416582+0000)
, 2022-04-03T01:17:31.738+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:17:31.840+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.030 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:42.420+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:42.420772+0000)
, 2022-04-03T01:17:42.850+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:17:42.876+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.031 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:53.846+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:17:53.890+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:17:54.128+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.031 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:04.860+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:18:05.148+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:05.148649+0000)
, 2022-04-03T01:18:05.189+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:05.189856+0000)

2022-04-03T14:13:10.032 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:15.638+0000 7fe007a1a700 -1 osd.7 64 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:18:15.853+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:18:16.150+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:16.151221+0000)

2022-04-03T14:13:10.033 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:27.152+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:27.153396+0000)
, 2022-04-03T01:18:27.187+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:18:27.193+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:27.194596+0000)

2022-04-03T14:13:10.033 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:37.492+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:18:37.788+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:18:37.893+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.034 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:48.447+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:48.448188+0000)
, 2022-04-03T01:18:48.613+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:18:48.759+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:59.623+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:18:59.846+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:18:59.894+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.036 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:10.861+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:19:10.920+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:19:11.162+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:11.162925+0000)

2022-04-03T14:13:10.036 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:21.772+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:19:21.819+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:19:21.935+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.037 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:32.900+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:19:32.907+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:19:33.166+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:33.167852+0000)

2022-04-03T14:13:10.038 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:43.790+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:19:43.864+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:19:44.008+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.038 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:55.171+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:55.172598+0000)
, 2022-04-03T01:19:55.173+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:19:55.213+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:55.213612+0000)

2022-04-03T14:13:10.039 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:06.215+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:06.215837+0000)
, 2022-04-03T01:20:06.274+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:06.274865+0000)
, 2022-04-03T01:20:06.304+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.040 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:17.176+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:17.177455+0000)
, 2022-04-03T01:20:17.191+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:20:17.216+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:17.218072+0000)

2022-04-03T14:13:10.040 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:28.248+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:20:28.283+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:28.283707+0000)
, 2022-04-03T01:20:28.421+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.041 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:39.288+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:20:39.470+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:39.470798+0000)
, 2022-04-03T01:20:39.494+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:39.494963+0000)

2022-04-03T14:13:10.042 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:50.292+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:50.292781+0000)
, 2022-04-03T01:20:50.436+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:20:50.475+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:50.475506+0000)

2022-04-03T14:13:10.042 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:01.432+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:21:01.478+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:01.479528+0000)
, 2022-04-03T01:21:01.503+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:01.504849+0000)

2022-04-03T14:13:10.043 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:12.514+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:21:12.642+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:21:12.764+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.044 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:23.513+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:23.514220+0000)
, 2022-04-03T01:21:23.614+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:21:23.662+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:34.704+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:21:34.761+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:21:34.778+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:45.519+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:21:45.522+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:45.524021+0000)
, 2022-04-03T01:21:45.640+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.046 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:56.614+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:21:56.932+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:21:56.943+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.047 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:07.836+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:22:07.858+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:22:08.201+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:08.202280+0000)

2022-04-03T14:13:10.047 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:18.631+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:22:18.779+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:22:18.896+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.048 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:29.691+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:22:29.796+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:22:30.205+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:30.207124+0000)

2022-04-03T14:13:10.049 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:41.209+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:41.209574+0000)
, 2022-04-03T01:22:41.249+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:41.249890+0000)
, 2022-04-03T01:22:41.336+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:41.337440+0000)

2022-04-03T14:13:10.049 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:51.778+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:22:52.211+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:52.212013+0000)
, 2022-04-03T01:22:52.251+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:52.252261+0000)

2022-04-03T14:13:10.050 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:03.213+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:03.214530+0000)
, 2022-04-03T01:23:03.253+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:03.254604+0000)
, 2022-04-03T01:23:03.304+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:14.037+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:23:14.213+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:23:14.216+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:14.216772+0000)

2022-04-03T14:13:10.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:25.259+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:25.259314+0000)
, 2022-04-03T01:23:25.343+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:23:25.355+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:25.355807+0000)

2022-04-03T14:13:10.052 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:36.360+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:36.360587+0000)
, 2022-04-03T01:23:36.387+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:23:36.430+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:47.263+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:47.264142+0000)
, 2022-04-03T01:23:47.364+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:47.365243+0000)
, 2022-04-03T01:23:47.547+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:47.548457+0000)

2022-04-03T14:13:10.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:58.432+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:23:58.454+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:23:58.552+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:58.552939+0000)

2022-04-03T14:13:10.054 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:09.557+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:09.557502+0000)
, 2022-04-03T01:24:09.582+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:09.582302+0000)
, 2022-04-03T01:24:09.893+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.055 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:20.382+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:24:20.561+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:20.562470+0000)
, 2022-04-03T01:24:20.585+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:20.586789+0000)

2022-04-03T14:13:10.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:31.566+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:31.567293+0000)
, 2022-04-03T01:24:31.590+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:31.591161+0000)
, 2022-04-03T01:24:31.878+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:42.903+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:24:43.030+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:24:43.107+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:53.600+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:53.600575+0000)
, 2022-04-03T01:24:53.754+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:24:54.115+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:04.710+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:25:05.091+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:25:05.240+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:15.608+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:15.609686+0000)
, 2022-04-03T01:25:15.794+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:25:16.197+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:27.246+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:27.246452+0000)
, 2022-04-03T01:25:27.262+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:25:27.285+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:27.285594+0000)

2022-04-03T14:13:10.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:38.248+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:38.248929+0000)
, 2022-04-03T01:25:38.287+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:38.287897+0000)
, 2022-04-03T01:25:38.350+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:49.091+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:25:49.250+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:49.251354+0000)
, 2022-04-03T01:25:49.289+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:49.290303+0000)

2022-04-03T14:13:10.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:00.253+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:00.253975+0000)
, 2022-04-03T01:26:00.292+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:00.292861+0000)
, 2022-04-03T01:26:00.356+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:11.350+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:26:11.413+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:26:11.425+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:11.425380+0000)

2022-04-03T14:13:10.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:22.345+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:26:22.378+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:26:22.428+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:22.429777+0000)

2022-04-03T14:13:10.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:33.357+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:26:33.433+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:33.434493+0000)
, 2022-04-03T01:26:33.576+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:44.493+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:26:44.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:26:44.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:10.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:55.524+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:26:55.627+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:55.627363+0000)
, 2022-04-03T01:26:55.651+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:55.651261+0000)

2022-04-03T14:13:10.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:06.626+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:27:06.630+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:06.631719+0000)
, 2022-04-03T01:27:06.654+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:06.655721+0000)

2022-04-03T14:13:10.066 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:17.451+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:17.452561+0000)
, 2022-04-03T01:27:17.551+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:27:17.635+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:17.636026+0000)

2022-04-03T14:13:10.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:28.576+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:27:28.639+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:28.640392+0000)
, 2022-04-03T01:27:28.663+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:28.664873+0000)

2022-04-03T14:13:10.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:39.669+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:39.669563+0000)
, 2022-04-03T01:27:40.008+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:27:40.144+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.068 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:50.648+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:50.649223+0000)
, 2022-04-03T01:27:50.672+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:50.674256+0000)
, 2022-04-03T01:27:51.116+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:01.678+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:01.678919+0000)
, 2022-04-03T01:28:02.051+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:28:02.160+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:13.038+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:13.209+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:13.213+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.070 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:23.964+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:28:24.027+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:24.111+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.071 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:35.028+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:28:35.118+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:35.196+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:45.984+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:46.020+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:46.100+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:57.235+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:57.292+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:57.292874+0000)
, 2022-04-03T01:28:57.319+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:08.224+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:29:08.294+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:08.295419+0000)
, 2022-04-03T01:29:08.333+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:08.334097+0000)

2022-04-03T14:13:10.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:19.175+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:29:19.241+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:29:19.297+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:19.297970+0000)

2022-04-03T14:13:10.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:30.201+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:29:30.299+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:30.300392+0000)
, 2022-04-03T01:29:30.337+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:30.338434+0000)

2022-04-03T14:13:10.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:41.340+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:41.340801+0000)
, 2022-04-03T01:29:41.512+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:41.512904+0000)
, 2022-04-03T01:29:41.696+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:41.696634+0000)

2022-04-03T14:13:10.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:52.304+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:52.305143+0000)
, 2022-04-03T01:29:52.341+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:52.342998+0000)
, 2022-04-03T01:29:52.516+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:52.517495+0000)

2022-04-03T14:13:10.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:03.344+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:03.345314+0000)
, 2022-04-03T01:30:03.521+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:03.522018+0000)
, 2022-04-03T01:30:03.705+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:03.705849+0000)

2022-04-03T14:13:10.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:14.526+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:14.526700+0000)
, 2022-04-03T01:30:14.710+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:14.710595+0000)
, 2022-04-03T01:30:14.735+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:14.735572+0000)

2022-04-03T14:13:10.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:25.531+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:25.531188+0000)
, 2022-04-03T01:30:25.714+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:25.715230+0000)
, 2022-04-03T01:30:25.739+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:25.740283+0000)

2022-04-03T14:13:10.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:36.718+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:36.719900+0000)
, 2022-04-03T01:30:36.743+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:36.744878+0000)
, 2022-04-03T01:30:37.031+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.079 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:47.540+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:47.540803+0000)
, 2022-04-03T01:30:47.723+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:47.724643+0000)
, 2022-04-03T01:30:47.749+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:47.750095+0000)

2022-04-03T14:13:10.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:58.728+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:58.729190+0000)
, 2022-04-03T01:30:58.754+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:58.754896+0000)
, 2022-04-03T01:30:58.892+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:09.942+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:31:09.968+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:31:09.970+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:20.763+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:20.764049+0000)
, 2022-04-03T01:31:20.774+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:31:20.801+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.082 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:31.817+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:31:31.830+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:31:31.862+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:42.844+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:31:43.329+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:43.329561+0000)
, 2022-04-03T01:31:43.367+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:43.367815+0000)

2022-04-03T14:13:10.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:53.776+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:53.777747+0000)
, 2022-04-03T01:31:53.797+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:31:54.330+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:54.331886+0000)

2022-04-03T14:13:10.084 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:04.890+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:32:05.333+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:05.334249+0000)
, 2022-04-03T01:32:05.371+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:05.372403+0000)

2022-04-03T14:13:10.085 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:16.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:32:16.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:32:16.335+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:16.336606+0000)

2022-04-03T14:13:10.085 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:27.338+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:27.339052+0000)
, 2022-04-03T01:32:27.377+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:27.377456+0000)
, 2022-04-03T01:32:27.475+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.086 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:38.378+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:38.379911+0000)
, 2022-04-03T01:32:38.519+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:32:38.586+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:38.587708+0000)

2022-04-03T14:13:10.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:49.342+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:49.343595+0000)
, 2022-04-03T01:32:49.381+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:49.382468+0000)
, 2022-04-03T01:32:49.487+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:00.384+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:00.384873+0000)
, 2022-04-03T01:33:00.526+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:00.596+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:00.596476+0000)

2022-04-03T14:13:10.088 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:11.601+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:11.601114+0000)
, 2022-04-03T01:33:11.785+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:11.785660+0000)
, 2022-04-03T01:33:11.811+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:11.811615+0000)

2022-04-03T14:13:10.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:22.526+0000 7fe007a1a700 -1 osd.7 65 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:22.604+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:22.605282+0000)
, 2022-04-03T01:33:22.789+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:22.790591+0000)

2022-04-03T14:13:10.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:33.265 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at
, 2022-04-03T01:33:33.352+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:33.353520+0000)
, 2022-04-03T01:33:33.391+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:33.391827+0000)

2022-04-03T14:13:10.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:44.355+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:44.355970+0000)
, 2022-04-03T01:33:44.393+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:44.394226+0000)
, 2022-04-03T01:33:44.613+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:44.613935+0000)

2022-04-03T14:13:10.091 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:55.357+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:55.358386+0000)
, 2022-04-03T01:33:55.395+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:55.396409+0000)
, 2022-04-03T01:33:55.617+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:55.618434+0000)

2022-04-03T14:13:10.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:06.397+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:06.398620+0000)
, 2022-04-03T01:34:06.622+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:06.623000+0000)
, 2022-04-03T01:34:06.791+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)

2022-04-03T14:13:10.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:17.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:34:17.399+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:17.400897+0000)
, 2022-04-03T01:34:17.627+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:17.627671+0000)

2022-04-03T14:13:10.093 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:28.461+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:34:28.632+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:28.632425+0000)
, 2022-04-03T01:34:28.818+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:28.818430+0000)

2022-04-03T14:13:10.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:39.822+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:39.823366+0000)
, 2022-04-03T01:34:39.848+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:39.849691+0000)
, 2022-04-03T01:34:40.017+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)

2022-04-03T14:13:10.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:50.641+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:50.642162+0000)
, 2022-04-03T01:34:50.826+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:50.828168+0000)
, 2022-04-03T01:34:50.853+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:50.854492+0000)

2022-04-03T14:13:10.095 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:01.832+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:01.833310+0000)
, 2022-04-03T01:35:01.859+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:01.859637+0000)
, 2022-04-03T01:35:02.086+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)

2022-04-03T14:13:10.096 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:13.031+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T01:35:13.159+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:35:13.232+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.096 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:23.867+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:23.868477+0000)
, 2022-04-03T01:35:24.006+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T01:35:24.152+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:35.034+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T01:35:35.082+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:35:35.284+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:46.074+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:35:46.173+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:35:46.379+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:57.090+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:35:57.279+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:35:57.385+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:57.386756+0000)

2022-04-03T14:13:10.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:08.236+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:36:08.377+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:36:08.388+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:08.389721+0000)

2022-04-03T14:13:10.100 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:19.260+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:36:19.307+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:36:19.391+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:19.392354+0000)

2022-04-03T14:13:10.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:30.394+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:30.395082+0000)
, 2022-04-03T01:36:30.395+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:36:30.429+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:30.429663+0000)

2022-04-03T14:13:10.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:41.508+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:36:41.611+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:36:41.687+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:41.688571+0000)

2022-04-03T14:13:10.102 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:52.433+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:52.434518+0000)
, 2022-04-03T01:36:52.561+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:36:52.581+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:03.697+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:03.698019+0000)
, 2022-04-03T01:37:03.718+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:37:03.723+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:14.702+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:14.702745+0000)
, 2022-04-03T01:37:14.767+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:37:14.808+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.104 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:25.706+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:25.707296+0000)
, 2022-04-03T01:37:25.738+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:37:25.783+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:36.788+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:37:36.771+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:37:36.905+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:36.906086+0000)

2022-04-03T14:13:10.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:47.763+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:37:47.910+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:47.910469+0000)
, 2022-04-03T01:37:47.932+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:47.932999+0000)

2022-04-03T14:13:10.106 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:58.831+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:37:58.835+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:37:58.914+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:58.915053+0000)

2022-04-03T14:13:10.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:09.918+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:09.919700+0000)
, 2022-04-03T01:38:09.942+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:09.942956+0000)
, 2022-04-03T01:38:10.184+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:20.803+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:20.923+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:20.924413+0000)
, 2022-04-03T01:38:20.946+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:20.947745+0000)

2022-04-03T14:13:10.108 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:31.950+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:31.952+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:31.952479+0000)
, 2022-04-03T01:38:32.114+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:43.117+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:43.426+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:43.427459+0000)
, 2022-04-03T01:38:43.457+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:43.458719+0000)

2022-04-03T14:13:10.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:53.960+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:53.961606+0000)
, 2022-04-03T01:38:54.319+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:54.429+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:54.430246+0000)

2022-04-03T14:13:10.110 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:05.254+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:39:05.432+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:05.432942+0000)
, 2022-04-03T01:39:05.462+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:05.463197+0000)

2022-04-03T14:13:10.111 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:16.435+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:16.435695+0000)
, 2022-04-03T01:39:16.465+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:16.465459+0000)
, 2022-04-03T01:39:16.713+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:27.437+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:27.438236+0000)
, 2022-04-03T01:39:27.466+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:27.467718+0000)
, 2022-04-03T01:39:27.676+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:38.468+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:38.469910+0000)
, 2022-04-03T01:39:38.585+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:39:38.763+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:38.764311+0000)

2022-04-03T14:13:10.113 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:49.437+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:39:49.442+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:49.442999+0000)
, 2022-04-03T01:39:49.471+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:49.472198+0000)

2022-04-03T14:13:10.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:00.474+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:00.474704+0000)
, 2022-04-03T01:40:00.536+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:40:00.774+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:00.774419+0000)

2022-04-03T14:13:10.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:11.694+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:40:11.778+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:11.779259+0000)
, 2022-04-03T01:40:11.971+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:11.972432+0000)

2022-04-03T14:13:10.115 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:22.732+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:40:22.783+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:22.784117+0000)
, 2022-04-03T01:40:22.812+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.731+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:40:33.788+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:33.789064+0000)
, 2022-04-03T01:40:33.981+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:33.981802+0000)

2022-04-03T14:13:10.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.801+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:40:44.985+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:44.986646+0000)
, 2022-04-03T01:40:45.008+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:45.009417+0000)

2022-04-03T14:13:10.118 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:55.966+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:40:55.990+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:55.991430+0000)
, 2022-04-03T01:40:56.013+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:56.014247+0000)

2022-04-03T14:13:10.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:06.994+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:06.995966+0000)
, 2022-04-03T01:41:07.018+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:07.019356+0000)
, 2022-04-03T01:41:07.051+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:18.023+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:18.024187+0000)
, 2022-04-03T01:41:18.186+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:41:18.201+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:29.004+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:29.005583+0000)
, 2022-04-03T01:41:29.028+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:29.029056+0000)
, 2022-04-03T01:41:29.179+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:40.249+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:41:40.406+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:41:40.469+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:40.469897+0000)

2022-04-03T14:13:10.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:51.038+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:51.039018+0000)
, 2022-04-03T01:41:51.258+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:41:51.472+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:51.472639+0000)

2022-04-03T14:13:10.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:02.242+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:42:02.463+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:42:02.474+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:02.475005+0000)

2022-04-03T14:13:10.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:13.476+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:13.477407+0000)
, 2022-04-03T01:42:13.484+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:42:13.502+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:13.503045+0000)

2022-04-03T14:13:10.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:24.479+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:24.479624+0000)
, 2022-04-03T01:42:24.484+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:42:24.504+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:24.505431+0000)

2022-04-03T14:13:10.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:35.481+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:35.482147+0000)
, 2022-04-03T01:42:35.507+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:35.507965+0000)
, 2022-04-03T01:42:35.748+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:46.509+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:46.510422+0000)
, 2022-04-03T01:42:46.709+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:42:46.831+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:57.527+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:42:57.564+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:42:57.849+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:57.849927+0000)

2022-04-03T14:13:10.127 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:08.572+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:43:08.854+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:08.854667+0000)
, 2022-04-03T01:43:08.995+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.127 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:19.859+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:19.859491+0000)
, 2022-04-03T01:43:20.021+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:43:20.051+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:20.052140+0000)

2022-04-03T14:13:10.128 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:30.656+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:43:30.862+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:30.863819+0000)
, 2022-04-03T01:43:30.958+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.129 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:41.984+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:43:42.061+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:42.061944+0000)
, 2022-04-03T01:43:42.086+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:42.086985+0000)

2022-04-03T14:13:10.130 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:52.872+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:52.873219+0000)
, 2022-04-03T01:43:53.065+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:53.066161+0000)
, 2022-04-03T01:43:53.077+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.130 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:04.070+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:04.070842+0000)
, 2022-04-03T01:44:04.096+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:04.096419+0000)
, 2022-04-03T01:44:04.130+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.131 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:15.074+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:44:15.100+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:15.101374+0000)
, 2022-04-03T01:44:15.131+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:26.105+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:26.105911+0000)
, 2022-04-03T01:44:26.207+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:44:26.275+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:37.268+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:44:37.304+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:44:37.443+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.133 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:48.279+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:44:48.367+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:44:48.511+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:48.511667+0000)

2022-04-03T14:13:10.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:59.119+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:59.120077+0000)
, 2022-04-03T01:44:59.292+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:44:59.513+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:59.514128+0000)

2022-04-03T14:13:10.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:10.506+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:45:10.515+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:10.516576+0000)
, 2022-04-03T01:45:10.541+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:10.541812+0000)

2022-04-03T14:13:10.135 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:21.153+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:45:21.287+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:45:21.513+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.136 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:32.296+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:45:32.437+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:45:32.520+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:32.521480+0000)

2022-04-03T14:13:10.137 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:43.523+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:43.523832+0000)
, 2022-04-03T01:45:43.530+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:45:43.548+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:43.548959+0000)

2022-04-03T14:13:10.137 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:54.484+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:45:54.525+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:54.526370+0000)
, 2022-04-03T01:45:54.550+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:54.551275+0000)

2022-04-03T14:13:10.138 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:05.553+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:05.553489+0000)
, 2022-04-03T01:46:05.727+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:46:05.837+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.139 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:16.672+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:46:16.748+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:46:16.935+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:16.936856+0000)

2022-04-03T14:13:10.139 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:27.557+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:27.558147+0000)
, 2022-04-03T01:46:27.592+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:46:27.639+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.140 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:38.599+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:46:38.608+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:46:38.946+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:38.946467+0000)

2022-04-03T14:13:10.141 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:49.703+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:46:49.951+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:49.951500+0000)
, 2022-04-03T01:46:50.096+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.141 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:00.564+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:00.565368+0000)
, 2022-04-03T01:47:00.763+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:47:00.955+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:00.955951+0000)

2022-04-03T14:13:10.142 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:11.960+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:11.960932+0000)
, 2022-04-03T01:47:12.127+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:47:12.152+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:12.152975+0000)

2022-04-03T14:13:10.143 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_ceph-c_cc9ed509562d767c1a5d8fc115593bc1453dfea1/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients
,     timeout=timeout)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 84, in __exit__

2022-04-03T14:13:10.144 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:26.570+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:26.571078+0000)
, 2022-04-03T01:47:26.778+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:47:26.967+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:26.967402+0000)

2022-04-03T14:13:10.144 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:37.758+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:47:37.971+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:37.972148+0000)
, 2022-04-03T01:47:38.126+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.145 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:48.976+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:48.977024+0000)
, 2022-04-03T01:47:49.024+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:47:49.168+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:49.169121+0000)

2022-04-03T14:13:10.146 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:59.972+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:47:59.981+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:59.981782+0000)
, 2022-04-03T01:48:00.104+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.146 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:11.117+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:48:11.148+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:48:11.178+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:11.178779+0000)

2022-04-03T14:13:10.147 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:21.984+0000 7fe007a1a700 -1 osd.7 66 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:48:21.989+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:21.990543+0000)
, 2022-04-03T01:48:22.127+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.148 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:32.993+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:32.995125+0000)
, 2022-04-03T01:48:33.188+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:33.188771+0000)
, 2022-04-03T01:48:33.212+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.148 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:43.587+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:43.588047+0000)
, 2022-04-03T01:48:43.895+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:48:43.999+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:43.999902+0000)

2022-04-03T14:13:10.149 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:54.589+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:54.590216+0000)
, 2022-04-03T01:48:54.942+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:48:54.977+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.150 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:05.882+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:49:05.983+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:49:06.009+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:06.009803+0000)

2022-04-03T14:13:10.151 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:17.014+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:17.014802+0000)
, 2022-04-03T01:49:17.081+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:49:17.087+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.151 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:28.019+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:28.019646+0000)
, 2022-04-03T01:49:28.019+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:49:28.055+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.152 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:39.036+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:49:39.116+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T01:49:39.130+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.153 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:50.135+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:49:50.220+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:50.222200+0000)
, 2022-04-03T01:49:50.232+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.154 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:01.033+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:01.034269+0000)
, 2022-04-03T01:50:01.078+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:50:01.226+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:01.227259+0000)

2022-04-03T14:13:10.154 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:12.236+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:50:12.257+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:12.257516+0000)
, 2022-04-03T01:50:12.265+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.155 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:23.090+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:50:23.236+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:23.237172+0000)
, 2022-04-03T01:50:23.261+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:23.262420+0000)

2022-04-03T14:13:10.156 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:34.170+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:50:34.241+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:34.242415+0000)
, 2022-04-03T01:50:34.266+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:34.267413+0000)

2022-04-03T14:13:10.156 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:45.247+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:45.247361+0000)
, 2022-04-03T01:50:45.271+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:45.272300+0000)
, 2022-04-03T01:50:45.591+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:45.591361+0000)

2022-04-03T14:13:10.157 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:56.252+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:56.252608+0000)
, 2022-04-03T01:50:56.277+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:56.277445+0000)
, 2022-04-03T01:50:56.593+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:56.593901+0000)

2022-04-03T14:13:10.158 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:07.281+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:07.282255+0000)
, 2022-04-03T01:51:07.595+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:07.596474+0000)
, 2022-04-03T01:51:07.618+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:07.619483+0000)

2022-04-03T14:13:10.158 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:18.598+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:18.598842+0000)
, 2022-04-03T01:51:18.621+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:18.621849+0000)
, 2022-04-03T01:51:18.871+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.159 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:29.291+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:29.291896+0000)
, 2022-04-03T01:51:29.601+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:29.601574+0000)
, 2022-04-03T01:51:29.623+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:29.624169+0000)

2022-04-03T14:13:10.160 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:40.625+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:40.626663+0000)
, 2022-04-03T01:51:40.841+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:51:40.848+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.161 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:51.707+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:51:51.954+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:51:52.026+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.161 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:02.608+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:02.608979+0000)
, 2022-04-03T01:52:02.631+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:02.631555+0000)
, 2022-04-03T01:52:02.970+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.162 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:13.752+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:52:14.040+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:52:14.091+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:14.091829+0000)

2022-04-03T14:13:10.163 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:24.640+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:52:25.087+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:52:25.095+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:25.096636+0000)

2022-04-03T14:13:10.164 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:36.100+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:36.101271+0000)
, 2022-04-03T01:52:36.116+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:52:36.140+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.164 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:47.105+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:47.106097+0000)
, 2022-04-03T01:52:47.112+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:52:47.300+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:47.301208+0000)

2022-04-03T14:13:10.165 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:58.043+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:52:58.110+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:58.110793+0000)
, 2022-04-03T01:52:58.305+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:58.305788+0000)

2022-04-03T14:13:10.166 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:09.184+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:53:09.309+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:09.310582+0000)
, 2022-04-03T01:53:09.333+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:09.335237+0000)

2022-04-03T14:13:10.166 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:20.274+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:53:20.314+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:20.315464+0000)
, 2022-04-03T01:53:20.339+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:20.340391+0000)

2022-04-03T14:13:10.167 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:31.182+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:53:31.320+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:31.320558+0000)
, 2022-04-03T01:53:31.345+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:31.345409+0000)

2022-04-03T14:13:10.168 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:42.349+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:42.350195+0000)
, 2022-04-03T01:53:42.377+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:53:42.629+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:42.630899+0000)

2022-04-03T14:13:10.169 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:53.328+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:53.329944+0000)
, 2022-04-03T01:53:53.353+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:53.354766+0000)
, 2022-04-03T01:53:53.409+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.169 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:04.358+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:04.359245+0000)
, 2022-04-03T01:54:04.562+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:54:04.635+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:04.635920+0000)

2022-04-03T14:13:10.170 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:15.534+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:54:15.637+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:15.638266+0000)
, 2022-04-03T01:54:15.659+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:15.659891+0000)

2022-04-03T14:13:10.171 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:26.598+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:54:26.639+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:26.640786+0000)
, 2022-04-03T01:54:26.661+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:26.662387+0000)

2022-04-03T14:13:10.171 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:37.642+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:37.643240+0000)
, 2022-04-03T01:54:37.663+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:37.664723+0000)
, 2022-04-03T01:54:37.872+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.172 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:48.666+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:48.667117+0000)
, 2022-04-03T01:54:48.907+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:54:48.968+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.173 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:59.647+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:59.647963+0000)
, 2022-04-03T01:54:59.669+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:59.669390+0000)
, 2022-04-03T01:54:59.815+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.174 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:10.806+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:55:10.822+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:55:10.931+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.174 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:21.802+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:55:21.909+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:55:22.174+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:55:22.175518+0000)

2022-04-03T14:13:10.175 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:32.740+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:55:32.757+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:55:32.933+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.176 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:43.753+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:55:43.840+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:55:44.184+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:55:44.185465+0000)

2022-04-03T14:13:10.177 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:54.739+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:55:54.955+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:55:55.188+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:55:55.190306+0000)

2022-04-03T14:13:10.177 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:05.780+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:56:06.194+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:06.195067+0000)
, 2022-04-03T01:56:06.389+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:06.390673+0000)

2022-04-03T14:13:10.178 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:17.199+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:17.200053+0000)
, 2022-04-03T01:56:17.395+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:17.395404+0000)
, 2022-04-03T01:56:17.419+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:17.419850+0000)

2022-04-03T14:13:10.179 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:28.203+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:28.205206+0000)
, 2022-04-03T01:56:28.398+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:28.400114+0000)
, 2022-04-03T01:56:28.405+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.179 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:39.273+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:56:39.404+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:39.405239+0000)
, 2022-04-03T01:56:39.428+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:39.429632+0000)

2022-04-03T14:13:10.180 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:50.399+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:56:50.409+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:50.409957+0000)
, 2022-04-03T01:56:50.433+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:50.434394+0000)

2022-04-03T14:13:10.181 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:01.220+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:01.220634+0000)
, 2022-04-03T01:57:01.395+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:57:01.414+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:01.414781+0000)

2022-04-03T14:13:10.181 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:12.418+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:12.419543+0000)
, 2022-04-03T01:57:12.442+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:12.443964+0000)
, 2022-04-03T01:57:12.676+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:12.677622+0000)

2022-04-03T14:13:10.182 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:23.448+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:23.448799+0000)
, 2022-04-03T01:57:23.679+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:23.680319+0000)
, 2022-04-03T01:57:23.699+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:23.700481+0000)

2022-04-03T14:13:10.183 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:34.429+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:34.429413+0000)
, 2022-04-03T01:57:34.453+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:34.453788+0000)
, 2022-04-03T01:57:34.682+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:34.683099+0000)

2022-04-03T14:13:10.184 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:45.458+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:45.458909+0000)
, 2022-04-03T01:57:45.685+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:45.685934+0000)
, 2022-04-03T01:57:45.704+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:45.705018+0000)

2022-04-03T14:13:10.184 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:56.462+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:56.463690+0000)
, 2022-04-03T01:57:56.687+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:56.688438+0000)
, 2022-04-03T01:57:56.705+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:56.707274+0000)

2022-04-03T14:13:10.185 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:07.690+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:07.690817+0000)
, 2022-04-03T01:58:07.709+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:07.709967+0000)
, 2022-04-03T01:58:07.924+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.186 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:18.711+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:18.712312+0000)
, 2022-04-03T01:58:18.992+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:58:19.015+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.186 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:29.694+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:29.695746+0000)
, 2022-04-03T01:58:29.713+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:29.714611+0000)
, 2022-04-03T01:58:30.009+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.187 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:40.962+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:58:41.028+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:58:41.075+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.188 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:52.002+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:58:52.054+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:58:52.268+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:52.269231+0000)

2022-04-03T14:13:10.189 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:02.788+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:59:03.120+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:59:03.214+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.189 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:14.237+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:59:14.277+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:14.279217+0000)
, 2022-04-03T01:59:14.285+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.190 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:25.256+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T01:59:25.282+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:25.284259+0000)
, 2022-04-03T01:59:25.322+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.191 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:36.288+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:36.289206+0000)
, 2022-04-03T01:59:36.428+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:59:36.485+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:36.485599+0000)

2022-04-03T14:13:10.191 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:47.293+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:47.293999+0000)
, 2022-04-03T01:59:47.490+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:47.490625+0000)
, 2022-04-03T01:59:47.513+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:47.513877+0000)

2022-04-03T14:13:10.192 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:58.389+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T01:59:58.494+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:58.495435+0000)
, 2022-04-03T01:59:58.518+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:58.519162+0000)

2022-04-03T14:13:10.193 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:09.499+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:09.500936+0000)
, 2022-04-03T02:00:09.523+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:09.524505+0000)
, 2022-04-03T02:00:09.598+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.194 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:20.529+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:20.529449+0000)
, 2022-04-03T02:00:20.631+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:00:20.722+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:20.722849+0000)

2022-04-03T14:13:10.194 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:31.510+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:31.511257+0000)
, 2022-04-03T02:00:31.533+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:31.534217+0000)
, 2022-04-03T02:00:31.560+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.195 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:42.567+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:00:42.727+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:42.728148+0000)
, 2022-04-03T02:00:42.741+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:42.742985+0000)

2022-04-03T14:13:10.196 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:53.730+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:53.730737+0000)
, 2022-04-03T02:00:53.744+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:53.745410+0000)
, 2022-04-03T02:00:53.976+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.197 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:04.632+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:04.732+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:04.733221+0000)
, 2022-04-03T02:01:04.747+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:04.747890+0000)

2022-04-03T14:13:10.197 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:15.734+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:15.735776+0000)
, 2022-04-03T02:01:15.749+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:15.750262+0000)
, 2022-04-03T02:01:16.002+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.198 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:26.737+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:26.738663+0000)
, 2022-04-03T02:01:26.751+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:26.752687+0000)
, 2022-04-03T02:01:26.986+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.199 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:37.792+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:37.802+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:38.247+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.199 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:48.980+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:49.349+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:49.349924+0000)
, 2022-04-03T02:01:49.392+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.200 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:59.777+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:01:59.988+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:02:00.353+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:00.355010+0000)

2022-04-03T14:13:10.201 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:11.359+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:11.360424+0000)
, 2022-04-03T02:02:11.557+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:11.557671+0000)
, 2022-04-03T02:02:11.578+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:11.578721+0000)

2022-04-03T14:13:10.201 WARNING:scrape:Ignoring malparsed backtrace: paramiko.buffered_pipe.PipeTimeout
, 
, During handling of the above exception, another exception occurred:

2022-04-03T14:13:10.202 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:27.752+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:27.752556+0000)
, 2022-04-03T02:02:27.764+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:02:27.766+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:27.766202+0000)

2022-04-03T14:13:10.203 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:38.754+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:38.755004+0000)
, 2022-04-03T02:02:38.767+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:38.768651+0000)
, 2022-04-03T02:02:39.054+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.204 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:49.769+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:49.770994+0000)
, 2022-04-03T02:02:50.119+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:02:50.348+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.204 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:00.773+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:00.773417+0000)
, 2022-04-03T02:03:00.866+0000 7fe007a1a700 -1 osd.7 67 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:01.256+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.205 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:12.264+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:12.387+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:12.387935+0000)
, 2022-04-03T02:03:12.458+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.206 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:23.392+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:23.392943+0000)
, 2022-04-03T02:03:23.492+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:03:23.579+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.206 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:33.684 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at
, 2022-04-03T02:03:33.766+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:33.767589+0000)
, 2022-04-03T02:03:33.780+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:33.780678+0000)

2022-04-03T14:13:10.207 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:44.769+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:44.770090+0000)
, 2022-04-03T02:03:44.782+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:44.783038+0000)
, 2022-04-03T02:03:45.103+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.208 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:55.890 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:03:55.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:03:55.986+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.209 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:06.821+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:04:06.925+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:04:07.411+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:07.412591+0000)

2022-04-03T14:13:10.209 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:17.851+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:04:18.416+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:18.417941+0000)
, 2022-04-03T02:04:18.431+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.210 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:28.792+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:28.792324+0000)
, 2022-04-03T02:04:28.825+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:04:29.421+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:29.422277+0000)

2022-04-03T14:13:10.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:40.426+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:40.427233+0000)
, 2022-04-03T02:04:40.493+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:04:40.534+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:51.431+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:51.432392+0000)
, 2022-04-03T02:04:51.540+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:04:51.631+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:51.632063+0000)

2022-04-03T14:13:10.212 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:02.319+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:05:02.435+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:02.437242+0000)
, 2022-04-03T02:05:02.469+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.213 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:13.442+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:13.442391+0000)
, 2022-04-03T02:05:13.593+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:05:13.642+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:13.642293+0000)

2022-04-03T14:13:10.214 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:24.646+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:24.647058+0000)
, 2022-04-03T02:05:24.665+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:24.666144+0000)
, 2022-04-03T02:05:24.785+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.214 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:35.651+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:35.652171+0000)
, 2022-04-03T02:05:35.670+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:35.671368+0000)
, 2022-04-03T02:05:35.699+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.215 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:46.675+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:46.675713+0000)
, 2022-04-03T02:05:46.776+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:05:46.797+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:46.797531+0000)

2022-04-03T14:13:10.216 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:57.680+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:57.680658+0000)
, 2022-04-03T02:05:57.786+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:05:57.799+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:57.799869+0000)

2022-04-03T14:13:10.216 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:08.800+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:08.802204+0000)
, 2022-04-03T02:06:08.812+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:08.813692+0000)
, 2022-04-03T02:06:08.816+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.217 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:19.778+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:06:19.803+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:19.804592+0000)
, 2022-04-03T02:06:19.815+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:19.815922+0000)

2022-04-03T14:13:10.218 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:30.714+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:06:30.806+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:30.807115+0000)
, 2022-04-03T02:06:30.818+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:30.818440+0000)

2022-04-03T14:13:10.219 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:41.819+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:41.820778+0000)
, 2022-04-03T02:06:41.849+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:06:41.906+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.219 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:52.822+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:52.823091+0000)
, 2022-04-03T02:06:52.869+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:06:53.240+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.220 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:03.813+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:03.814457+0000)
, 2022-04-03T02:07:03.824+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:03.825350+0000)
, 2022-04-03T02:07:03.924+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.221 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:14.827+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:14.827702+0000)
, 2022-04-03T02:07:14.864+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:07:15.273+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.221 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:26.316+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:07:26.502+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:26.503720+0000)
, 2022-04-03T02:07:26.513+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.222 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:36.831+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:36.832450+0000)
, 2022-04-03T02:07:37.436+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:07:37.508+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:37.508833+0000)

2022-04-03T14:13:10.223 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:48.338+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:07:48.513+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:48.513625+0000)
, 2022-04-03T02:07:48.595+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.224 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:58.992+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:07:59.290+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:07:59.518+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:59.518496+0000)

2022-04-03T14:13:10.224 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:10.522+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:10.523380+0000)
, 2022-04-03T02:08:10.624+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:08:10.722+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:10.723121+0000)

2022-04-03T14:13:10.225 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:21.527+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:21.528236+0000)
, 2022-04-03T02:08:21.645+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:08:21.726+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:21.728147+0000)

2022-04-03T14:13:10.226 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:32.533+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:32.533405+0000)
, 2022-04-03T02:08:32.550+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:08:32.699+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.226 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:43.608+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:08:43.680+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:08:43.737+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:43.738257+0000)

2022-04-03T14:13:10.227 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:54.742+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:54.743644+0000)
, 2022-04-03T02:08:54.760+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:54.761556+0000)
, 2022-04-03T02:08:54.839+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:54.840356+0000)

2022-04-03T14:13:10.228 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:05.713+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:09:05.748+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:05.748602+0000)
, 2022-04-03T02:09:05.766+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:05.766691+0000)

2022-04-03T14:13:10.229 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:16.753+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:16.753817+0000)
, 2022-04-03T02:09:16.771+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:16.772148+0000)
, 2022-04-03T02:09:16.845+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:16.845359+0000)

2022-04-03T14:13:10.229 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:27.758+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:27.759237+0000)
, 2022-04-03T02:09:27.776+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:27.777431+0000)
, 2022-04-03T02:09:27.846+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:27.847901+0000)

2022-04-03T14:13:10.230 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:38.782+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:38.782889+0000)
, 2022-04-03T02:09:38.849+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:38.850297+0000)
, 2022-04-03T02:09:38.858+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:38.858757+0000)

2022-04-03T14:13:10.231 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:49.852+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:49.852853+0000)
, 2022-04-03T02:09:49.860+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:49.861429+0000)
, 2022-04-03T02:09:50.017+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.231 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:00.792+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:00.792890+0000)
, 2022-04-03T02:10:00.855+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:00.855369+0000)
, 2022-04-03T02:10:00.863+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:00.863809+0000)

2022-04-03T14:13:10.232 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:11.865+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:11.866024+0000)
, 2022-04-03T02:10:12.098+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:12.293+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.233 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:23.009+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:23.212+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:10:23.241+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.233 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:33.870+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:33.870728+0000)
, 2022-04-03T02:10:34.016+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:34.075+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.234 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:44.968+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:45.109+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:45.450+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.235 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:56.411+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:10:56.454+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:10:56.598+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:56.599137+0000)

2022-04-03T14:13:10.235 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:07.072+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:11:07.474+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:11:07.551+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.236 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:18.454+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:11:18.497+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:11:18.609+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:18.609453+0000)

2022-04-03T14:13:10.237 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:28.881+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:28.882416+0000)
, 2022-04-03T02:11:29.475+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:11:29.505+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.238 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:40.474+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:11:40.618+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:40.619168+0000)
, 2022-04-03T02:11:40.820+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:40.821004+0000)

2022-04-03T14:13:10.238 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:51.623+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:51.624288+0000)
, 2022-04-03T02:11:51.825+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:51.826349+0000)
, 2022-04-03T02:11:51.844+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:51.844736+0000)

2022-04-03T14:13:10.239 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:02.628+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:02.629171+0000)
, 2022-04-03T02:12:02.629+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:12:02.676+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:13.738+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:12:13.743+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:12:13.835+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:13.837095+0000)

2022-04-03T14:13:10.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:24.841+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:24.842420+0000)
, 2022-04-03T02:12:24.858+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:24.859259+0000)
, 2022-04-03T02:12:24.886+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:24.886863+0000)

2022-04-03T14:13:10.241 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:35.847+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:35.847625+0000)
, 2022-04-03T02:12:35.863+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:35.863688+0000)
, 2022-04-03T02:12:35.888+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:35.889161+0000)

2022-04-03T14:13:10.242 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:46.857+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:12:46.868+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:46.868964+0000)
, 2022-04-03T02:12:46.891+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:46.891507+0000)

2022-04-03T14:13:10.243 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:57.893+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:57.893931+0000)
, 2022-04-03T02:12:57.901+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:57.901880+0000)
, 2022-04-03T02:12:58.144+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.243 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:08.878+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:08.879285+0000)
, 2022-04-03T02:13:08.895+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:08.896313+0000)
, 2022-04-03T02:13:08.903+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:08.904271+0000)

2022-04-03T14:13:10.244 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:19.898+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:19.898610+0000)
, 2022-04-03T02:13:19.906+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:19.906712+0000)
, 2022-04-03T02:13:20.287+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.245 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:30.887+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:30.889295+0000)
, 2022-04-03T02:13:30.899+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:30.901113+0000)
, 2022-04-03T02:13:30.908+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:30.909165+0000)

2022-04-03T14:13:10.245 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:41.910+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:41.911493+0000)
, 2022-04-03T02:13:42.116+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:13:42.358+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.246 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:52.913+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:52.914079+0000)
, 2022-04-03T02:13:53.335+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:13:53.680+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:53.680910+0000)

2022-04-03T14:13:10.247 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:03.916+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:03.916607+0000)
, 2022-04-03T02:14:03.965+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:14:04.300+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.248 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:14.978+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:14:15.236+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:14:15.684+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.248 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:26.663+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:14:26.696+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:26.696844+0000)
, 2022-04-03T02:14:26.849+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.249 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:37.402+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:14:37.689+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:14:37.701+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:37.701633+0000)

2022-04-03T14:13:10.250 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:48.701+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:14:48.706+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:48.706650+0000)
, 2022-04-03T02:14:48.755+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.250 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:59.408+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:14:59.710+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:59.711693+0000)
, 2022-04-03T02:14:59.719+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.251 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:10.726+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:15:10.796+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:10.921+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:10.921899+0000)

2022-04-03T14:13:10.252 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:21.897+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:21.925+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:21.925577+0000)
, 2022-04-03T02:15:21.926+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:21.926868+0000)

2022-04-03T14:13:10.252 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:32.871+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:15:32.905+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:32.926+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:32.927994+0000)

2022-04-03T14:13:10.253 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:43.904+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:43.929+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:43.930366+0000)
, 2022-04-03T02:15:43.936+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:43.937279+0000)

2022-04-03T14:13:10.254 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:54.940+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:54.942245+0000)
, 2022-04-03T02:15:54.949+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:54.957+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:54.957632+0000)

2022-04-03T14:13:10.255 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:05.941+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:05.942044+0000)
, 2022-04-03T02:16:05.946+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:05.947288+0000)
, 2022-04-03T02:16:05.962+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:05.962938+0000)

2022-04-03T14:13:10.255 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:16.943+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:16.944571+0000)
, 2022-04-03T02:16:16.951+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:16.952725+0000)
, 2022-04-03T02:16:16.966+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:16.968272+0000)

2022-04-03T14:13:10.256 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:27.973+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:27.973826+0000)
, 2022-04-03T02:16:28.280+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:16:28.289+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.257 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:38.962+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:38.962884+0000)
, 2022-04-03T02:16:38.978+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:38.978898+0000)
, 2022-04-03T02:16:39.408+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.257 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:49.983+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:49.984032+0000)
, 2022-04-03T02:16:50.493+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:16:50.500+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.258 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:00.972+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:00.973198+0000)
, 2022-04-03T02:17:00.987+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:00.988983+0000)
, 2022-04-03T02:17:01.551+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.259 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:12.473+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:17:12.578+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:12.773+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:12.773926+0000)

2022-04-03T14:13:10.260 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:23.600+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:23.770+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:23.778+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:23.779098+0000)

2022-04-03T14:13:10.260 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:34.410+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:17:34.637+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:34.783+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:34.784024+0000)

2022-04-03T14:13:10.261 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:45.788+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:45.789165+0000)
, 2022-04-03T02:17:45.844+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:45.936+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.262 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:56.948+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:56.959+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:56.959630+0000)
, 2022-04-03T02:17:56.964+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:56.965286+0000)

2022-04-03T14:13:10.262 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:07.754+0000 7fe007a1a700 -1 osd.7 68 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:18:07.799+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:07.799449+0000)
, 2022-04-03T02:18:07.962+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:07.962308+0000)

2022-04-03T14:13:10.263 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:18.803+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:18.804048+0000)
, 2022-04-03T02:18:18.963+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:18.964748+0000)
, 2022-04-03T02:18:18.969+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:18.970190+0000)

2022-04-03T14:13:10.264 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:29.966+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:29.967268+0000)
, 2022-04-03T02:18:29.971+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:29.972701+0000)
, 2022-04-03T02:18:30.018+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:30.018779+0000)

2022-04-03T14:13:10.265 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:40.574+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:18:40.613+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:18:40.663+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.265 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:51.672+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:18:51.775+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:18:51.819+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:51.819424+0000)

2022-04-03T14:13:10.266 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:02.566+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:19:02.584+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:19:02.807+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.267 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:13.687+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:19:13.708+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:19:13.828+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:13.829590+0000)

2022-04-03T14:13:10.267 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:24.834+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:24.834575+0000)
, 2022-04-03T02:19:24.979+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:24.979360+0000)
, 2022-04-03T02:19:24.984+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:24.984597+0000)

2022-04-03T14:13:10.268 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:35.839+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:35.839697+0000)
, 2022-04-03T02:19:35.862+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:19:35.981+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:35.981911+0000)

2022-04-03T14:13:10.269 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:46.843+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:46.844645+0000)
, 2022-04-03T02:19:46.982+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:46.984253+0000)
, 2022-04-03T02:19:46.988+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:46.989303+0000)

2022-04-03T14:13:10.270 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:57.990+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:57.991446+0000)
, 2022-04-03T02:19:58.061+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:58.062083+0000)
, 2022-04-03T02:19:58.073+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:58.073865+0000)

2022-04-03T14:13:10.270 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:08.988+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:08.989188+0000)
, 2022-04-03T02:20:08.993+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:08.993769+0000)
, 2022-04-03T02:20:09.067+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:09.067526+0000)

2022-04-03T14:13:10.271 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:19.994+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:19.995906+0000)
, 2022-04-03T02:20:20.071+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:20.072531+0000)
, 2022-04-03T02:20:20.083+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:20.084553+0000)

2022-04-03T14:13:10.272 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:30.993+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:30.994271+0000)
, 2022-04-03T02:20:30.997+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:30.998266+0000)
, 2022-04-03T02:20:31.076+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:31.077831+0000)

2022-04-03T14:13:10.272 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:42.082+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:42.083247+0000)
, 2022-04-03T02:20:42.094+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:42.095445+0000)
, 2022-04-03T02:20:42.397+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.273 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:53.100+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:53.100526+0000)
, 2022-04-03T02:20:53.401+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:20:53.674+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.274 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:04.104+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:04.105799+0000)
, 2022-04-03T02:21:04.132+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:04.610+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.274 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:15.127+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:15.603+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:21:15.739+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.275 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:26.652+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:21:26.890+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:26.890970+0000)
, 2022-04-03T02:21:26.907+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.276 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:37.594+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:21:37.629+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:37.895+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:37.896004+0000)

2022-04-03T14:13:10.277 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:48.707+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:48.900+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:48.901320+0000)
, 2022-04-03T02:21:48.934+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.277 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:59.906+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:59.906881+0000)
, 2022-04-03T02:21:59.970+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:22:00.013+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:00.014212+0000)

2022-04-03T14:13:10.278 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:10.897+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:22:10.911+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:10.912182+0000)
, 2022-04-03T02:22:11.016+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:11.016603+0000)

2022-04-03T14:13:10.279 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:21.916+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:21.917322+0000)
, 2022-04-03T02:22:22.017+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:22.018908+0000)
, 2022-04-03T02:22:22.022+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:22.023465+0000)

2022-04-03T14:13:10.279 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:32.915+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:22:32.921+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:32.922449+0000)
, 2022-04-03T02:22:33.020+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:33.021446+0000)

2022-04-03T14:13:10.280 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:43.934+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:22:44.023+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:44.024036+0000)
, 2022-04-03T02:22:44.027+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:44.028115+0000)

2022-04-03T14:13:10.281 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:55.030+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:55.030363+0000)
, 2022-04-03T02:22:55.146+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:55.147001+0000)
, 2022-04-03T02:22:55.159+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:55.159973+0000)

2022-04-03T14:13:10.282 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:06.027+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:06.028872+0000)
, 2022-04-03T02:23:06.031+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:06.032622+0000)
, 2022-04-03T02:23:06.150+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:06.151880+0000)

2022-04-03T14:13:10.282 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:17.033+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:17.035206+0000)
, 2022-04-03T02:23:17.135+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:23:17.155+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:17.157313+0000)

2022-04-03T14:13:10.283 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:28.162+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:28.163098+0000)
, 2022-04-03T02:23:28.175+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:28.176190+0000)
, 2022-04-03T02:23:28.649+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.284 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:38.849+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:23:38.883+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:23:38.951+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:38.951952+0000)

2022-04-03T14:13:10.284 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:49.935+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:23:49.953+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:49.955270+0000)
, 2022-04-03T02:23:50.036+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:50.037563+0000)

2022-04-03T14:13:10.285 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:01.037+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:01.039287+0000)
, 2022-04-03T02:24:01.041+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:01.042780+0000)
, 2022-04-03T02:24:01.060+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.286 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:12.140+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:24:12.178+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:12.178402+0000)
, 2022-04-03T02:24:12.191+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:12.191381+0000)

2022-04-03T14:13:10.287 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:23.194+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:23.194809+0000)
, 2022-04-03T02:24:23.250+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:24:23.795+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.287 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:34.230+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:24:34.792+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:24:34.814+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.288 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:45.749+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:24:45.856+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:24:45.857+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.289 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:56.908+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:24:56.976+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:56.976388+0000)
, 2022-04-03T02:24:57.048+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:57.048372+0000)

2022-04-03T14:13:10.289 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:07.980+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:07.981229+0000)
, 2022-04-03T02:25:08.049+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:08.050826+0000)
, 2022-04-03T02:25:08.052+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:08.053534+0000)

2022-04-03T14:13:10.290 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:19.051+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:19.053236+0000)
, 2022-04-03T02:25:19.054+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:19.055813+0000)
, 2022-04-03T02:25:19.145+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.291 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:30.055+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:30.055716+0000)
, 2022-04-03T02:25:30.057+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:30.058217+0000)
, 2022-04-03T02:25:30.106+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.291 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:41.060+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:41.060526+0000)
, 2022-04-03T02:25:41.214+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:41.214762+0000)
, 2022-04-03T02:25:41.226+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:41.226719+0000)

2022-04-03T14:13:10.292 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:52.218+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:52.219848+0000)
, 2022-04-03T02:25:52.230+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:52.231908+0000)
, 2022-04-03T02:25:52.410+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.293 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:03.126+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:26:03.223+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:03.224868+0000)
, 2022-04-03T02:26:03.236+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:03.237206+0000)

2022-04-03T14:13:10.293 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:14.229+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:14.229962+0000)
, 2022-04-03T02:26:14.242+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:14.242880+0000)
, 2022-04-03T02:26:14.324+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.294 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:25.433+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:26:25.671+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:26:26.019+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:26.019352+0000)

2022-04-03T14:13:10.295 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:36.252+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:36.253589+0000)
, 2022-04-03T02:26:36.438+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:26:36.643+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.296 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:47.465+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:26:47.605+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:26:47.753+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.296 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:58.074+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:58.074556+0000)
, 2022-04-03T02:26:58.076+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:58.076385+0000)
, 2022-04-03T02:26:58.251+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:58.251391+0000)

2022-04-03T14:13:10.297 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:09.076+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:09.077993+0000)
, 2022-04-03T02:27:09.253+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:09.255042+0000)
, 2022-04-03T02:27:09.267+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:09.268596+0000)

2022-04-03T14:13:10.298 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:20.257+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:20.258664+0000)
, 2022-04-03T02:27:20.271+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:20.272348+0000)
, 2022-04-03T02:27:20.392+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.298 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:31.274+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:31.275873+0000)
, 2022-04-03T02:27:31.378+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:27:31.397+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.299 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:42.511+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:27:42.882+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:27:42.938+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.300 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:53.622+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:27:53.783+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:27:54.051+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:54.052550+0000)

2022-04-03T14:13:10.301 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:04.779+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:28:05.054+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:05.056130+0000)
, 2022-04-03T02:28:05.084+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:05.085358+0000)

2022-04-03T14:13:10.301 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:16.058+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:16.059749+0000)
, 2022-04-03T02:28:16.085+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:16.087102+0000)
, 2022-04-03T02:28:16.085+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:16.087073+0000)

2022-04-03T14:13:10.302 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:27.089+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:27.089356+0000)
, 2022-04-03T02:28:27.281+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:27.282116+0000)
, 2022-04-03T02:28:27.293+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:27.294296+0000)

2022-04-03T14:13:10.303 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:38.286+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:38.287497+0000)
, 2022-04-03T02:28:38.298+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:38.299786+0000)
, 2022-04-03T02:28:38.575+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.303 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:49.304+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:49.305065+0000)
, 2022-04-03T02:28:49.698+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:28:49.707+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.304 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:00.298+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:00.298912+0000)
, 2022-04-03T02:29:00.310+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:00.310348+0000)
, 2022-04-03T02:29:00.694+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.305 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:11.661+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:29:11.809+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:29:11.834+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.305 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:22.320+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:22.321484+0000)
, 2022-04-03T02:29:22.781+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:29:22.784+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.306 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:33.737+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:29:33.812+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:33.955+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:44.882+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:29:45.019+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:45.044+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:55.779+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:29:55.904+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:56.012+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.308 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:06.911+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:30:07.021+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:30:07.110+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:07.111756+0000)

2022-04-03T14:13:10.309 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:18.100+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:30:18.114+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:18.114614+0000)
, 2022-04-03T02:30:18.114+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:18.114678+0000)

2022-04-03T14:13:10.310 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:28.804+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:30:29.087+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:30:29.116+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:29.117179+0000)

2022-04-03T14:13:10.310 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:40.118+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:40.119820+0000)
, 2022-04-03T02:30:40.118+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:40.119799+0000)
, 2022-04-03T02:30:40.125+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:40.126872+0000)

2022-04-03T14:13:10.311 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:51.121+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:51.122434+0000)
, 2022-04-03T02:30:51.131+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:51.132018+0000)
, 2022-04-03T02:30:51.292+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:02.124+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:02.124955+0000)
, 2022-04-03T02:31:02.124+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:02.125024+0000)
, 2022-04-03T02:31:02.136+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:02.137220+0000)

2022-04-03T14:13:10.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:13.142+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:13.142575+0000)
, 2022-04-03T02:31:13.365+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:13.366029+0000)
, 2022-04-03T02:31:13.373+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:13.374030+0000)

2022-04-03T14:13:10.313 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:24.129+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:24.130169+0000)
, 2022-04-03T02:31:24.146+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:24.147664+0000)
, 2022-04-03T02:31:24.370+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:24.371622+0000)

2022-04-03T14:13:10.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:35.151+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:35.153110+0000)
, 2022-04-03T02:31:35.376+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:35.377088+0000)
, 2022-04-03T02:31:35.382+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:35.384035+0000)

2022-04-03T14:13:10.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:46.382+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:46.382320+0000)
, 2022-04-03T02:31:46.388+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:46.389173+0000)
, 2022-04-03T02:31:46.541+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:57.379+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:31:57.386+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:57.387868+0000)
, 2022-04-03T02:31:57.393+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:57.394899+0000)

2022-04-03T14:13:10.316 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:08.398+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:08.399666+0000)
, 2022-04-03T02:32:08.444+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:32:08.587+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:19.470+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:32:19.529+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:32:20.142+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:20.143206+0000)

2022-04-03T14:13:10.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:30.409+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:30.409877+0000)
, 2022-04-03T02:32:30.446+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:32:30.588+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.318 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:41.651+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:32:42.087+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:32:42.147+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:42.148462+0000)

2022-04-03T14:13:10.319 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:53.075+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:32:53.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:32:53.130 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:10.320 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:03.746+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:33:03.987+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:33:04.153+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:04.153878+0000)

2022-04-03T14:13:10.320 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:14.870+0000 7fe007a1a700 -1 osd.7 69 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:33:15.156+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:15.156642+0000)
, 2022-04-03T02:33:15.156+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:15.156642+0000)

2022-04-03T14:13:10.321 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:25.993+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:33:26.125+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:33:26.158+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:26.159463+0000)

2022-04-03T14:13:10.322 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:36.434+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:36.434728+0000)
, 2022-04-03T02:33:36.440+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:36.441009+0000)
, 2022-04-03T02:33:36.807+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.322 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:47.445+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:47.446303+0000)
, 2022-04-03T02:33:47.862+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:33:47.989+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.323 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:58.451+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:58.451707+0000)
, 2022-04-03T02:33:58.689+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:33:58.868+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.324 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:09.830+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:09.877+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:34:10.169+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:10.170447+0000)

2022-04-03T14:13:10.324 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:20.999+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:34:21.083+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:34:21.172+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:21.173162+0000)

2022-04-03T14:13:10.325 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:31.937+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:34:31.947+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:32.052+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.326 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:43.099+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:34:43.177+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:43.178807+0000)
, 2022-04-03T02:34:43.177+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:43.178807+0000)

2022-04-03T14:13:10.327 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:53.995+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:34:54.096+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:34:54.180+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:54.181315+0000)

2022-04-03T14:13:10.327 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:04.979+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:35:05.183+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:05.184240+0000)
, 2022-04-03T02:35:05.183+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:05.184241+0000)

2022-04-03T14:13:10.328 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:16.186+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:16.187077+0000)
, 2022-04-03T02:35:16.186+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:16.187078+0000)
, 2022-04-03T02:35:16.208+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.329 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:27.111+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:35:27.188+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:27.189890+0000)
, 2022-04-03T02:35:27.188+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:27.189890+0000)

2022-04-03T14:13:10.329 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:38.191+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:38.192548+0000)
, 2022-04-03T02:35:38.191+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:38.192548+0000)
, 2022-04-03T02:35:38.267+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:38.268086+0000)

2022-04-03T14:13:10.330 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:49.246+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:35:49.272+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:49.272992+0000)
, 2022-04-03T02:35:49.435+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:00.197+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:00.197777+0000)
, 2022-04-03T02:36:00.278+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:00.278281+0000)
, 2022-04-03T02:36:00.280+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:11.282+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:11.283577+0000)
, 2022-04-03T02:36:11.512+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:11.512965+0000)
, 2022-04-03T02:36:11.517+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:11.517880+0000)

2022-04-03T14:13:10.332 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:22.517+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:22.518528+0000)
, 2022-04-03T02:36:22.522+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:22.523255+0000)
, 2022-04-03T02:36:22.649+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.333 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:33.295+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:36:33.523+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:33.524155+0000)
, 2022-04-03T02:36:33.528+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:33.528651+0000)

2022-04-03T14:13:10.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:44.528+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:44.529697+0000)
, 2022-04-03T02:36:44.533+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:44.534097+0000)
, 2022-04-03T02:36:44.677+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:55.534+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:55.535229+0000)
, 2022-04-03T02:36:55.538+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:55.539519+0000)
, 2022-04-03T02:36:55.683+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:06.544+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:06.545355+0000)
, 2022-04-03T02:37:06.702+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:06.808+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:17.657+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:17.718+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:18.109+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:28.693+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:28.697+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:28.699+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.337 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:39.735+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:40.123+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:37:40.186+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:51.005+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:37:51.098+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:37:51.219+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:51.220880+0000)

2022-04-03T14:13:10.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:01.849+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:38:02.048+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:38:02.126+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:13.225+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:13.226007+0000)
, 2022-04-03T02:38:13.225+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:13.226031+0000)
, 2022-04-03T02:38:13.235+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.340 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:24.227+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:24.228596+0000)
, 2022-04-03T02:38:24.254+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:38:24.348+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:24.349386+0000)

2022-04-03T14:13:10.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:35.163+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:38:35.231+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:35.231307+0000)
, 2022-04-03T02:38:35.231+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:35.231306+0000)

2022-04-03T14:13:10.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:46.233+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:46.233714+0000)
, 2022-04-03T02:38:46.233+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:46.233719+0000)
, 2022-04-03T02:38:46.358+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:46.359955+0000)

2022-04-03T14:13:10.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:57.235+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:57.236448+0000)
, 2022-04-03T02:38:57.235+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:57.236456+0000)
, 2022-04-03T02:38:57.364+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:57.365631+0000)

2022-04-03T14:13:10.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:08.238+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:08.239159+0000)
, 2022-04-03T02:39:08.370+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:08.370971+0000)
, 2022-04-03T02:39:08.603+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:08.603817+0000)

2022-04-03T14:13:10.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:19.376+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:19.376514+0000)
, 2022-04-03T02:39:19.491+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:39:19.609+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:19.609367+0000)

2022-04-03T14:13:10.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:30.380+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:30.381936+0000)
, 2022-04-03T02:39:30.383+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:39:30.613+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:30.614955+0000)

2022-04-03T14:13:10.346 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:41.593+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:39:41.619+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:41.620472+0000)
, 2022-04-03T02:39:41.623+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:41.623924+0000)

2022-04-03T14:13:10.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:52.628+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:52.629627+0000)
, 2022-04-03T02:39:52.646+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:39:53.027+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:03.631+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:03.631320+0000)
, 2022-04-03T02:40:03.634+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:03.635064+0000)
, 2022-04-03T02:40:03.680+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:14.635+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:14.636838+0000)
, 2022-04-03T02:40:14.639+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:14.640406+0000)
, 2022-04-03T02:40:15.104+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:25.641+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:25.642165+0000)
, 2022-04-03T02:40:25.644+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:25.645902+0000)
, 2022-04-03T02:40:26.257+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:26.258607+0000)

2022-04-03T14:13:10.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:36.651+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:36.651301+0000)
, 2022-04-03T02:40:37.260+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:37.260979+0000)
, 2022-04-03T02:40:37.260+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:37.261080+0000)

2022-04-03T14:13:10.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:48.263+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:48.263358+0000)
, 2022-04-03T02:40:48.263+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:48.263382+0000)
, 2022-04-03T02:40:48.281+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:58.710+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:40:59.264+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:59.265988+0000)
, 2022-04-03T02:40:59.264+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:59.266149+0000)

2022-04-03T14:13:10.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:10.267+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:10.268547+0000)
, 2022-04-03T02:41:10.430+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:10.430801+0000)
, 2022-04-03T02:41:10.548+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:21.426+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:41:21.435+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:21.436157+0000)
, 2022-04-03T02:41:21.491+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:32.272+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:32.273078+0000)
, 2022-04-03T02:41:32.411+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:41:32.440+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:32.441514+0000)

2022-04-03T14:13:10.354 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:43.445+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:43.446765+0000)
, 2022-04-03T02:41:43.609+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:41:43.680+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:43.681848+0000)

2022-04-03T14:13:10.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:54.620+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:41:54.638+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:41:54.663+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:05.457+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:05.457646+0000)
, 2022-04-03T02:42:05.507+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:42:05.537+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:16.462+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:16.463019+0000)
, 2022-04-03T02:42:16.481+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:42:16.697+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:16.698189+0000)

2022-04-03T14:13:10.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:27.460+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:42:27.467+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:27.468662+0000)
, 2022-04-03T02:42:27.701+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:38.535+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:42:38.560+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:42:38.709+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:38.709359+0000)

2022-04-03T14:13:10.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:49.713+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:42:49.714+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:49.714681+0000)
, 2022-04-03T02:42:49.719+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:49.719839+0000)

2022-04-03T14:13:10.359 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:00.689+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:43:00.718+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:00.719638+0000)
, 2022-04-03T02:43:00.724+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:00.725220+0000)

2022-04-03T14:13:10.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:11.724+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:11.725123+0000)
, 2022-04-03T02:43:11.729+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:11.731131+0000)
, 2022-04-03T02:43:12.294+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:12.295160+0000)

2022-04-03T14:13:10.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:22.960+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:43:23.297+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:23.297759+0000)
, 2022-04-03T02:43:23.297+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:23.297779+0000)

2022-04-03T14:13:10.361 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:33.741+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:33.742517+0000)
, 2022-04-03T02:43:33.978+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:43:34.299+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:34.300380+0000)

2022-04-03T14:13:10.362 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:44.987+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:43:45.302+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:45.303169+0000)
, 2022-04-03T02:43:45.302+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:45.303169+0000)

2022-04-03T14:13:10.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:56.305+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:56.305873+0000)
, 2022-04-03T02:43:56.320+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:43:56.357+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:07.127+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:44:07.308+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:07.308360+0000)
, 2022-04-03T02:44:07.308+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:07.308449+0000)

2022-04-03T14:13:10.364 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:18.222+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:44:18.310+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:18.310979+0000)
, 2022-04-03T02:44:18.310+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:18.310977+0000)

2022-04-03T14:13:10.365 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:29.156+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:44:29.222+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:44:29.312+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:29.313469+0000)

2022-04-03T14:13:10.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:40.315+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:40.315887+0000)
, 2022-04-03T02:44:40.315+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:40.315964+0000)
, 2022-04-03T02:44:40.463+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:51.318+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:51.318463+0000)
, 2022-04-03T02:44:51.467+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:44:51.539+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:51.539386+0000)

2022-04-03T14:13:10.367 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:02.320+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:02.320882+0000)
, 2022-04-03T02:45:02.320+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:02.320888+0000)
, 2022-04-03T02:45:02.423+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:13.461+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:45:13.549+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:13.550429+0000)
, 2022-04-03T02:45:13.724+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:24.555+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:24.555705+0000)
, 2022-04-03T02:45:24.792+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:24.793027+0000)
, 2022-04-03T02:45:24.799+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:24.799942+0000)

2022-04-03T14:13:10.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:35.405+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:45:35.560+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:35.561272+0000)
, 2022-04-03T02:45:35.797+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:35.798920+0000)

2022-04-03T14:13:10.370 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:46.565+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:46.566801+0000)
, 2022-04-03T02:45:46.804+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:46.804833+0000)
, 2022-04-03T02:45:46.810+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:46.811134+0000)

2022-04-03T14:13:10.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:57.570+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:57.572138+0000)
, 2022-04-03T02:45:57.809+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:57.810143+0000)
, 2022-04-03T02:45:57.816+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:57.816944+0000)

2022-04-03T14:13:10.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:08.815+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:08.816064+0000)
, 2022-04-03T02:46:08.822+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:08.822458+0000)
, 2022-04-03T02:46:09.052+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:19.826+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:19.827871+0000)
, 2022-04-03T02:46:20.127+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:46:20.212+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:30.825+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:30.827198+0000)
, 2022-04-03T02:46:30.832+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:30.833441+0000)
, 2022-04-03T02:46:31.144+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:42.044+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:46:42.048+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:46:42.241+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:53.118+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:46:53.212+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:46:53.346+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:53.346739+0000)

2022-04-03T14:13:10.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:04.007+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:47:04.108+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:47:04.247+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:15.063+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:47:15.278+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:47:15.351+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:15.352082+0000)

2022-04-03T14:13:10.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:26.354+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:26.354718+0000)
, 2022-04-03T02:47:26.354+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:26.354728+0000)
, 2022-04-03T02:47:26.399+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:37.314+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:47:37.316+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:47:37.356+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:37.357359+0000)

2022-04-03T14:13:10.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:48.315+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:47:48.358+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:48.359737+0000)
, 2022-04-03T02:47:48.358+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:48.359804+0000)

2022-04-03T14:13:10.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:59.274+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:47:59.300+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:47:59.361+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:59.362052+0000)

2022-04-03T14:13:10.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:10.364+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:10.364579+0000)
, 2022-04-03T02:48:10.385+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:48:10.639+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:10.639531+0000)

2022-04-03T14:13:10.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:21.591+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:48:21.643+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:21.644892+0000)
, 2022-04-03T02:48:21.883+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:21.884651+0000)

2022-04-03T14:13:10.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:32.446+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:48:32.649+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:32.650605+0000)
, 2022-04-03T02:48:32.756+0000 7fe007a1a700 -1 osd.7 70 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:43.371+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:43.371956+0000)
, 2022-04-03T02:48:43.419+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:48:43.608+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:54.528+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:48:54.568+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:48:54.661+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:54.661618+0000)

2022-04-03T14:13:10.383 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:05.376+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:05.377282+0000)
, 2022-04-03T02:49:05.499+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:49:05.537+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.383 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:16.378+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:16.379361+0000)
, 2022-04-03T02:49:16.539+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:49:16.672+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:16.673000+0000)

2022-04-03T14:13:10.384 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:27.678+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:27.678527+0000)
, 2022-04-03T02:49:27.732+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:49:27.830+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.385 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:38.675+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:49:38.683+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:38.684425+0000)
, 2022-04-03T02:49:38.700+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.385 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:49.689+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:49.690405+0000)
, 2022-04-03T02:49:49.749+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:49:49.766+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:00.602+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:50:00.634+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:50:00.695+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:00.696164+0000)

2022-04-03T14:13:10.387 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:11.701+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:11.701683+0000)
, 2022-04-03T02:50:11.775+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:50:11.941+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:11.941612+0000)

2022-04-03T14:13:10.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:22.815+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:50:22.946+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:22.947201+0000)
, 2022-04-03T02:50:22.949+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:22.950789+0000)

2022-04-03T14:13:10.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:33.782+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:50:33.806+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T02:50:33.952+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:33.952918+0000)

2022-04-03T14:13:10.389 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:44.940+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:50:44.956+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:44.958246+0000)
, 2022-04-03T02:50:44.961+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:44.961472+0000)

2022-04-03T14:13:10.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:56.047+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:50:56.401+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:56.401943+0000)
, 2022-04-03T02:50:56.401+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:56.401941+0000)

2022-04-03T14:13:10.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:06.971+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:06.971909+0000)
, 2022-04-03T02:51:06.998+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:51:07.385+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.391 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:18.065+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:51:18.405+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:18.406745+0000)
, 2022-04-03T02:51:18.405+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:18.406789+0000)

2022-04-03T14:13:10.392 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:28.981+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:28.981932+0000)
, 2022-04-03T02:51:29.056+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:51:29.408+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:29.409278+0000)

2022-04-03T14:13:10.392 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:40.410+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:40.411791+0000)
, 2022-04-03T02:51:40.410+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:40.411758+0000)
, 2022-04-03T02:51:40.600+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:51.413+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:51.414307+0000)
, 2022-04-03T02:51:51.413+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:51.414369+0000)
, 2022-04-03T02:51:51.650+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.394 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:02.259+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:52:02.416+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:02.416710+0000)
, 2022-04-03T02:52:02.416+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:02.416751+0000)

2022-04-03T14:13:10.394 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:13.418+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:13.419252+0000)
, 2022-04-03T02:52:13.419+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:13.419385+0000)
, 2022-04-03T02:52:13.744+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.395 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:24.458+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:52:24.765+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:24.766691+0000)
, 2022-04-03T02:52:24.999+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.396 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:35.423+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:35.424528+0000)
, 2022-04-03T02:52:35.574+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:52:35.771+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:35.772089+0000)

2022-04-03T14:13:10.396 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:46.580+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:52:46.777+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:46.777768+0000)
, 2022-04-03T02:52:47.032+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:47.032938+0000)

2022-04-03T14:13:10.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:58.039+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:58.039499+0000)
, 2022-04-03T02:52:58.039+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:58.039478+0000)
, 2022-04-03T02:52:58.068+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:08.787+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:08.788751+0000)
, 2022-04-03T02:53:09.003+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:53:09.044+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:09.045839+0000)

2022-04-03T14:13:10.399 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:20.051+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:20.052557+0000)
, 2022-04-03T02:53:20.051+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:20.052549+0000)
, 2022-04-03T02:53:20.051+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.399 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:30.799+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:30.800135+0000)
, 2022-04-03T02:53:31.038+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:53:31.058+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:31.059194+0000)

2022-04-03T14:13:10.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:42.064+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:42.065474+0000)
, 2022-04-03T02:53:42.064+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:42.065553+0000)
, 2022-04-03T02:53:42.263+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.401 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:53.071+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:53.071869+0000)
, 2022-04-03T02:53:53.224+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:53:53.443+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:53.443902+0000)

2022-04-03T14:13:10.401 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:04.083+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:04.083714+0000)
, 2022-04-03T02:54:04.090+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:54:04.195+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.402 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:15.088+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:15.088908+0000)
, 2022-04-03T02:54:15.337+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:54:15.449+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:15.449718+0000)

2022-04-03T14:13:10.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:26.451+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:26.452440+0000)
, 2022-04-03T02:54:26.485+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:54:26.612+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:37.454+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:37.455088+0000)
, 2022-04-03T02:54:37.454+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:37.455093+0000)
, 2022-04-03T02:54:37.596+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:48.457+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:48.458049+0000)
, 2022-04-03T02:54:48.554+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:54:48.592+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:59.582+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:54:59.583+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:54:59.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:10.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:10.626+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:55:10.635+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:55:10.810+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:21.772+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:55:21.806+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:55:21.853+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:21.854399+0000)

2022-04-03T14:13:10.407 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:32.625+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:55:32.755+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:55:32.797+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.408 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:43.668+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:55:43.752+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:55:43.864+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:43.865532+0000)

2022-04-03T14:13:10.408 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:54.870+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:54.871120+0000)
, 2022-04-03T02:55:54.929+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:55:55.135+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:55.136040+0000)

2022-04-03T14:13:10.409 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:05.722+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:05.856+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:05.876+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:05.876744+0000)

2022-04-03T14:13:10.410 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:16.818+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:16.882+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:16.882428+0000)
, 2022-04-03T02:56:17.147+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:17.147360+0000)

2022-04-03T14:13:10.410 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:28.152+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:28.153314+0000)
, 2022-04-03T02:56:28.156+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:28.157889+0000)
, 2022-04-03T02:56:28.481+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:28.482128+0000)

2022-04-03T14:13:10.411 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:39.048+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:39.157+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:39.159126+0000)
, 2022-04-03T02:56:39.162+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:39.163790+0000)

2022-04-03T14:13:10.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:50.164+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:50.164683+0000)
, 2022-04-03T02:56:50.169+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:50.169572+0000)
, 2022-04-03T02:56:50.486+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:50.487632+0000)

2022-04-03T14:13:10.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:01.170+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:01.170742+0000)
, 2022-04-03T02:57:01.175+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:01.175429+0000)
, 2022-04-03T02:57:01.201+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.413 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:12.179+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:12.180812+0000)
, 2022-04-03T02:57:12.321+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:57:12.491+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:12.492675+0000)

2022-04-03T14:13:10.414 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:23.305+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:57:23.494+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:23.494965+0000)
, 2022-04-03T02:57:23.494+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:23.495110+0000)

2022-04-03T14:13:10.414 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:34.247+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:57:34.299+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:57:34.497+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:34.497626+0000)

2022-04-03T14:13:10.415 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:45.321+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:57:45.499+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:45.499997+0000)
, 2022-04-03T02:57:45.499+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:45.500055+0000)

2022-04-03T14:13:10.416 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:56.501+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:56.502552+0000)
, 2022-04-03T02:57:56.779+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:57:56.931+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:56.932034+0000)

2022-04-03T14:13:10.416 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:07.504+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:07.504748+0000)
, 2022-04-03T02:58:07.504+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:07.504908+0000)
, 2022-04-03T02:58:07.876+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.417 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:18.506+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:18.507244+0000)
, 2022-04-03T02:58:18.853+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:58:18.921+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.418 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:29.789+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:58:29.946+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:29.947747+0000)
, 2022-04-03T02:58:30.013+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.418 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:40.729+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:58:40.952+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:40.953199+0000)
, 2022-04-03T02:58:41.004+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.419 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:51.950+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:58:51.958+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:51.958864+0000)
, 2022-04-03T02:58:51.961+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.420 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:02.604+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T02:59:02.780+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:59:02.825+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.421 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:13.763+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:59:13.868+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:59:13.969+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:13.970002+0000)

2022-04-03T14:13:10.421 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:24.975+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:24.975899+0000)
, 2022-04-03T02:59:25.057+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:59:25.244+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:25.244924+0000)

2022-04-03T14:13:10.422 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:35.981+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:35.981434+0000)
, 2022-04-03T02:59:36.015+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:59:36.105+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.423 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:46.986+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:46.986862+0000)
, 2022-04-03T02:59:47.147+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T02:59:47.257+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:47.257650+0000)

2022-04-03T14:13:10.423 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:58.262+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:58.263833+0000)
, 2022-04-03T02:59:58.265+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:58.266482+0000)
, 2022-04-03T02:59:58.528+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:58.529230+0000)

2022-04-03T14:13:10.424 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:09.110+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:00:09.268+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:09.269400+0000)
, 2022-04-03T03:00:09.271+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:09.272326+0000)

2022-04-03T14:13:10.425 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:20.275+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:20.275297+0000)
, 2022-04-03T03:00:20.277+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:20.278033+0000)
, 2022-04-03T03:00:20.533+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:20.534200+0000)

2022-04-03T14:13:10.425 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:31.282+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:31.283377+0000)
, 2022-04-03T03:00:31.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:00:31.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:10.426 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:42.287+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:42.289188+0000)
, 2022-04-03T03:00:42.345+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:00:42.529+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.427 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:53.294+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:53.295239+0000)
, 2022-04-03T03:00:53.541+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:53.542045+0000)
, 2022-04-03T03:00:53.541+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:53.542058+0000)

2022-04-03T14:13:10.427 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:04.298+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:04.299091+0000)
, 2022-04-03T03:01:04.300+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:04.300877+0000)
, 2022-04-03T03:01:04.544+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:04.544821+0000)

2022-04-03T14:13:10.428 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:15.385+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:01:15.546+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:15.547503+0000)
, 2022-04-03T03:01:15.546+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:15.547504+0000)

2022-04-03T14:13:10.429 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:26.548+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:26.550134+0000)
, 2022-04-03T03:01:26.966+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:01:26.980+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.429 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:37.552+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:37.552790+0000)
, 2022-04-03T03:01:37.571+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:01:37.950+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.430 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:48.555+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:48.555868+0000)
, 2022-04-03T03:01:48.848+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:01:48.992+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.431 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:59.919+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:02:00.036+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:00.054+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:00.055615+0000)

2022-04-03T14:13:10.431 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:10.872+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:10.883+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:02:11.060+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:11.061384+0000)

2022-04-03T14:13:10.432 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:21.968+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:22.066+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:22.067019+0000)
, 2022-04-03T03:02:22.191+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.433 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:32.856+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:32.955+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:02:33.071+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:33.072558+0000)

2022-04-03T14:13:10.433 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:44.039+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:02:44.076+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:44.077997+0000)
, 2022-04-03T03:02:44.197+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.434 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:55.273+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:55.319+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:02:55.364+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:55.365066+0000)

2022-04-03T14:13:10.435 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:06.187+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:03:06.227+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:03:06.336+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.435 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:17.203+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:03:17.369+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:03:17.379+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:17.380504+0000)

2022-04-03T14:13:10.436 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:28.387+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:28.387727+0000)
, 2022-04-03T03:03:28.387+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:28.387728+0000)
, 2022-04-03T03:03:28.485+0000 7fe007a1a700 -1 osd.7 71 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.437 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:38.595+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:03:38.973+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:03:39.105+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:39.105692+0000)

2022-04-03T14:13:10.437 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:49.826+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:03:50.022+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:03:50.111+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:50.111210+0000)

2022-04-03T14:13:10.438 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:00.960+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:04:01.115+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:01.116778+0000)
, 2022-04-03T03:04:01.185+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.439 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:11.991+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:04:12.121+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:12.122454+0000)
, 2022-04-03T03:04:12.194+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.440 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:23.127+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:23.127916+0000)
, 2022-04-03T03:04:23.346+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:04:23.427+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:23.427637+0000)

2022-04-03T14:13:10.440 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:33.898+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:04:34.132+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:34.133404+0000)
, 2022-04-03T03:04:34.365+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.441 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:45.137+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:45.138829+0000)
, 2022-04-03T03:04:45.214+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:04:45.441+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:45.442148+0000)

2022-04-03T14:13:10.442 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:56.400+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:04:56.447+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:56.449263+0000)
, 2022-04-03T03:04:56.447+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:56.449263+0000)

2022-04-03T14:13:10.442 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:07.149+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:07.149868+0000)
, 2022-04-03T03:05:07.429+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:05:07.457+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:07.457498+0000)

2022-04-03T14:13:10.443 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:18.271+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:05:18.465+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:18.466062+0000)
, 2022-04-03T03:05:18.465+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:18.466062+0000)

2022-04-03T14:13:10.444 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:29.453+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:05:29.473+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:29.474258+0000)
, 2022-04-03T03:05:29.473+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:29.474256+0000)

2022-04-03T14:13:10.444 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:40.481+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:40.482159+0000)
, 2022-04-03T03:05:40.481+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:40.482136+0000)
, 2022-04-03T03:05:40.600+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.445 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:51.490+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:51.490568+0000)
, 2022-04-03T03:05:51.611+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:05:51.613+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:51.614227+0000)

2022-04-03T14:13:10.446 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:02.615+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:02.616646+0000)
, 2022-04-03T03:06:02.615+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:02.616801+0000)
, 2022-04-03T03:06:02.623+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.446 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:13.617+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:13.619181+0000)
, 2022-04-03T03:06:13.617+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:13.619218+0000)
, 2022-04-03T03:06:13.713+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.447 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:24.621+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:24.622145+0000)
, 2022-04-03T03:06:24.809+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:06:24.847+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.448 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:35.623+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:35.625000+0000)
, 2022-04-03T03:06:35.623+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:35.625000+0000)
, 2022-04-03T03:06:35.926+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.448 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:46.626+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:46.627678+0000)
, 2022-04-03T03:06:46.858+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:06:47.039+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.449 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:58.012+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:06:58.060+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:06:58.182+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.450 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:08.950+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:07:09.046+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:07:09.062+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.451 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:20.054+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:07:20.138+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:07:20.142+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.451 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:31.097+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:07:31.206+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:07:31.222+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:31.223104+0000)

2022-04-03T14:13:10.452 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:42.044+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:07:42.228+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:42.228836+0000)
, 2022-04-03T03:07:42.278+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.453 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:53.080+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:07:53.234+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:53.234612+0000)
, 2022-04-03T03:07:53.568+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:53.568574+0000)

2022-04-03T14:13:10.453 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:04.087+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:08:04.089+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:08:04.239+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:04.240506+0000)

2022-04-03T14:13:10.454 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:15.096+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:08:15.245+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:15.246157+0000)
, 2022-04-03T03:08:15.581+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:15.582030+0000)

2022-04-03T14:13:10.455 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:26.588+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:26.588625+0000)
, 2022-04-03T03:08:26.588+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:26.588723+0000)
, 2022-04-03T03:08:26.651+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:26.651786+0000)

2022-04-03T14:13:10.455 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:37.434+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:08:37.594+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:37.595103+0000)
, 2022-04-03T03:08:37.594+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:37.595115+0000)

2022-04-03T14:13:10.456 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:48.600+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:48.601666+0000)
, 2022-04-03T03:08:48.600+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:48.601687+0000)
, 2022-04-03T03:08:48.655+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:48.656463+0000)

2022-04-03T14:13:10.457 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:59.608+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:59.608586+0000)
, 2022-04-03T03:08:59.658+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:59.659050+0000)
, 2022-04-03T03:08:59.658+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:59.659058+0000)

2022-04-03T14:13:10.457 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:10.614+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:10.615100+0000)
, 2022-04-03T03:09:10.614+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:10.615211+0000)
, 2022-04-03T03:09:10.661+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:10.661515+0000)

2022-04-03T14:13:10.458 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:21.621+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:21.622448+0000)
, 2022-04-03T03:09:21.662+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:21.663992+0000)
, 2022-04-03T03:09:21.662+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:21.664223+0000)

2022-04-03T14:13:10.459 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:32.665+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:32.666646+0000)
, 2022-04-03T03:09:32.665+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:32.666711+0000)
, 2022-04-03T03:09:32.924+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.459 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:43.668+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:43.669059+0000)
, 2022-04-03T03:09:43.668+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:43.669048+0000)
, 2022-04-03T03:09:43.791+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.460 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:54.766+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:09:54.960+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:09:55.109+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.461 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:05.672+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:05.674108+0000)
, 2022-04-03T03:10:05.672+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:05.674215+0000)
, 2022-04-03T03:10:05.938+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.461 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:16.675+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:16.676681+0000)
, 2022-04-03T03:10:16.887+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:10:17.284+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.462 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:28.313+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:28.313695+0000)
, 2022-04-03T03:10:28.429+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:10:28.469+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.463 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:39.109+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:10:39.251+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:10:39.318+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:39.319306+0000)

2022-04-03T14:13:10.463 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:50.162+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:10:50.324+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:50.324960+0000)
, 2022-04-03T03:10:50.467+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.464 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:01.329+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:01.330363+0000)
, 2022-04-03T03:11:01.436+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:11:01.672+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:01.672792+0000)

2022-04-03T14:13:10.465 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:12.335+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:12.336099+0000)
, 2022-04-03T03:11:12.443+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:11:12.677+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:12.678056+0000)

2022-04-03T14:13:10.465 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:23.340+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:23.341497+0000)
, 2022-04-03T03:11:23.566+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:11:23.682+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:23.684163+0000)

2022-04-03T14:13:10.466 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:34.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:11:34.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:11:34.603+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.467 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:45.472+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:11:45.685+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:11:45.695+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:45.695848+0000)

2022-04-03T14:13:10.467 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:56.699+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:56.699303+0000)
, 2022-04-03T03:11:56.699+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:56.700306+0000)
, 2022-04-03T03:11:56.705+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:56.705917+0000)

2022-04-03T14:13:10.468 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:07.700+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:07.701780+0000)
, 2022-04-03T03:12:07.700+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:07.701835+0000)
, 2022-04-03T03:12:07.705+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:07.706385+0000)

2022-04-03T14:13:10.469 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:18.703+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:18.704210+0000)
, 2022-04-03T03:12:18.712+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:18.712935+0000)
, 2022-04-03T03:12:18.717+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:18.718266+0000)

2022-04-03T14:13:10.470 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:29.718+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:29.719307+0000)
, 2022-04-03T03:12:29.724+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:29.724480+0000)
, 2022-04-03T03:12:29.838+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.470 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:40.724+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:40.725290+0000)
, 2022-04-03T03:12:40.729+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:40.730420+0000)
, 2022-04-03T03:12:40.880+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.471 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:51.735+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:51.736231+0000)
, 2022-04-03T03:12:51.812+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:12:52.177+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.472 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:02.741+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:02.742318+0000)
, 2022-04-03T03:13:03.180+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:03.196+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.472 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:13.748+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:13.748595+0000)
, 2022-04-03T03:13:13.800+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:13:14.121+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.473 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:25.097+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:25.154+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:25.403+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:25.404704+0000)

2022-04-03T14:13:10.473 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:35.760+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:35.761455+0000)
, 2022-04-03T03:13:36.125+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:36.191+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.474 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:46.981+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:47.196+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:47.226+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.475 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:58.372+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:13:58.421+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:58.422103+0000)
, 2022-04-03T03:13:58.727+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:58.727906+0000)

2022-04-03T14:13:10.475 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:09.072+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:14:09.245+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:14:09.427+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:09.428132+0000)

2022-04-03T14:13:10.476 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:20.374+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:14:20.432+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:20.434087+0000)
, 2022-04-03T03:14:20.724+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.477 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:31.439+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:31.439474+0000)
, 2022-04-03T03:14:31.617+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:14:31.734+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:31.735289+0000)

2022-04-03T14:13:10.478 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:42.444+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:42.445237+0000)
, 2022-04-03T03:14:42.564+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:14:42.736+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:42.737541+0000)

2022-04-03T14:13:10.478 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:53.466+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:14:53.738+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:53.739676+0000)
, 2022-04-03T03:14:53.738+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:53.739796+0000)

2022-04-03T14:13:10.479 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:04.740+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:15:04.740+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:04.742068+0000)
, 2022-04-03T03:15:04.740+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:04.742217+0000)

2022-04-03T14:13:10.480 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:15.461+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:15.462069+0000)
, 2022-04-03T03:15:15.744+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:15.744531+0000)
, 2022-04-03T03:15:15.744+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:15.744627+0000)

2022-04-03T14:13:10.480 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:26.746+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:26.747153+0000)
, 2022-04-03T03:15:26.747+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:15:26.816+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:26.817684+0000)

2022-04-03T14:13:10.481 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:37.748+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:37.749401+0000)
, 2022-04-03T03:15:37.748+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:37.749454+0000)
, 2022-04-03T03:15:37.790+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.482 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:48.751+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:48.751949+0000)
, 2022-04-03T03:15:48.829+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:48.830032+0000)
, 2022-04-03T03:15:48.833+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:48.834055+0000)

2022-04-03T14:13:10.482 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:59.835+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:59.836119+0000)
, 2022-04-03T03:15:59.839+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:59.839899+0000)
, 2022-04-03T03:15:59.959+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.483 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:10.841+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:10.842182+0000)
, 2022-04-03T03:16:10.844+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:10.845832+0000)
, 2022-04-03T03:16:10.899+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.484 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:21.851+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:21.851796+0000)
, 2022-04-03T03:16:21.966+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:16:22.204+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.484 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:33.002+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:16:33.102+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:16:33.500+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:33.500770+0000)

2022-04-03T14:13:10.485 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:43.862+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:43.863142+0000)
, 2022-04-03T03:16:44.021+0000 7fe007a1a700 -1 osd.7 72 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:16:44.204+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.486 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:55.341+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:16:55.412+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:16:55.470+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.486 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:06.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:17:06.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:17:06.493+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.487 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:17.213+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:17:17.524+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:17.524264+0000)
, 2022-04-03T03:17:17.548+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.488 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:28.528+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:28.530015+0000)
, 2022-04-03T03:17:28.631+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:17:28.772+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:28.773794+0000)

2022-04-03T14:13:10.488 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:39.388+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:17:39.535+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:39.535856+0000)
, 2022-04-03T03:17:39.730+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.489 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:50.541+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:50.541432+0000)
, 2022-04-03T03:17:50.708+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:17:50.778+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:50.779124+0000)

2022-04-03T14:13:10.490 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:01.773+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:18:01.781+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:01.781899+0000)
, 2022-04-03T03:18:01.781+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:01.781910+0000)

2022-04-03T14:13:10.491 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:12.716+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:18:12.783+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:12.784521+0000)
, 2022-04-03T03:18:12.783+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:12.784531+0000)

2022-04-03T14:13:10.491 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:23.786+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:23.787042+0000)
, 2022-04-03T03:18:23.786+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:23.787044+0000)
, 2022-04-03T03:18:23.914+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:23.914737+0000)

2022-04-03T14:13:10.492 WARNING:scrape:Ignoring malparsed backtrace:     -1> 2022-04-03T03:18:34.708+0000 7efccb1ce700 -1 MDSIOContextBase: failed with -110, restarting...
, 2022-04-03T03:18:34.708+0000 7efccc1d0700 -1 mds.0.purge_queue operator(): Error -110 loading Journaler
, 2022-04-03T03:18:34.745 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at

2022-04-03T14:13:10.493 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:45.442+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:18:45.446+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:18:45.479+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.493 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:56.429+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:18:56.575+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:56.576378+0000)
, 2022-04-03T03:18:56.794+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:56.795561+0000)

2022-04-03T14:13:10.494 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:07.456+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:19:07.486+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:19:07.581+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:07.582039+0000)

2022-04-03T14:13:10.495 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:18.534+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:19:18.587+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:18.587900+0000)
, 2022-04-03T03:19:18.800+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:18.800617+0000)

2022-04-03T14:13:10.495 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:29.663+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:19:29.801+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:29.803060+0000)
, 2022-04-03T03:19:29.801+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:29.803117+0000)

2022-04-03T14:13:10.496 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:40.638+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:19:40.804+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:40.805607+0000)
, 2022-04-03T03:19:40.804+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:40.805619+0000)

2022-04-03T14:13:10.497 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:51.807+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:51.808046+0000)
, 2022-04-03T03:19:51.807+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:51.808055+0000)
, 2022-04-03T03:19:51.962+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:51.963288+0000)

2022-04-03T14:13:10.497 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:02.810+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:02.810557+0000)
, 2022-04-03T03:20:02.968+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:02.969706+0000)
, 2022-04-03T03:20:02.969+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:02.970631+0000)

2022-04-03T14:13:10.498 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:13.812+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:13.813196+0000)
, 2022-04-03T03:20:13.817+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:20:13.975+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:13.975927+0000)

2022-04-03T14:13:10.499 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:24.981+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:24.982762+0000)
, 2022-04-03T03:20:24.981+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:24.982897+0000)
, 2022-04-03T03:20:25.184+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.499 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:35.989+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:35.989835+0000)
, 2022-04-03T03:20:36.037+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:20:36.423+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.500 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:46.936+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:20:46.993+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:46.994900+0000)
, 2022-04-03T03:20:46.993+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:46.994952+0000)

2022-04-03T14:13:10.501 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:58.062+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:20:58.381+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:58.639+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:58.639739+0000)

2022-04-03T14:13:10.501 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:09.006+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:09.007464+0000)
, 2022-04-03T03:21:09.187+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:21:09.343+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.502 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:20.170+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:21:20.399+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:20.651+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:20.651491+0000)

2022-04-03T14:13:10.503 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:31.413+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:31.656+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:31.657179+0000)
, 2022-04-03T03:21:31.767+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.503 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:42.261+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:42.662+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:42.662743+0000)
, 2022-04-03T03:21:42.812+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.504 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:53.668+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:53.668826+0000)
, 2022-04-03T03:21:53.835+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:53.835666+0000)
, 2022-04-03T03:21:53.835+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:53.835771+0000)

2022-04-03T14:13:10.505 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:04.799+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:22:04.836+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:22:04.836+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:04.838113+0000)

2022-04-03T14:13:10.505 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:15.680+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:15.681637+0000)
, 2022-04-03T03:22:15.818+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:22:15.839+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:15.840718+0000)

2022-04-03T14:13:10.506 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:26.842+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:26.842962+0000)
, 2022-04-03T03:22:26.842+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:26.842989+0000)
, 2022-04-03T03:22:26.971+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.507 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:37.863+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:22:37.942 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:22:37.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:10.507 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:48.846+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:48.847720+0000)
, 2022-04-03T03:22:49.062+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:22:49.074+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:49.075651+0000)

2022-04-03T14:13:10.508 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:00.016+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:23:00.083+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:00.084352+0000)
, 2022-04-03T03:23:00.083+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:00.084352+0000)

2022-04-03T14:13:10.509 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:10.977+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:23:11.091+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:11.091311+0000)
, 2022-04-03T03:23:11.091+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:11.091425+0000)

2022-04-03T14:13:10.509 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:22.098+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:22.098660+0000)
, 2022-04-03T03:23:22.098+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:22.098662+0000)
, 2022-04-03T03:23:22.290+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.510 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:33.111+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:23:33.158+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:23:33.291+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.511 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:44.112+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:44.113374+0000)
, 2022-04-03T03:23:44.163+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:23:44.287+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.512 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:55.362+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:23:55.381+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:23:55.440+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.512 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:06.377+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:24:06.381+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:24:06.739+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:06.740854+0000)

2022-04-03T14:13:10.513 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:17.157+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:24:17.376+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:24:17.399+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.514 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:28.314+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:24:28.752+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:28.753374+0000)
, 2022-04-03T03:24:28.758+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.514 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:39.196+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:24:39.405+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:24:39.759+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:39.759411+0000)

2022-04-03T14:13:10.515 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:50.485+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:24:50.764+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:50.765391+0000)
, 2022-04-03T03:24:50.873+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:50.874849+0000)

2022-04-03T14:13:10.516 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:01.770+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:01.771269+0000)
, 2022-04-03T03:25:01.876+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:01.877120+0000)
, 2022-04-03T03:25:01.876+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:01.877144+0000)

2022-04-03T14:13:10.516 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:12.730+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:25:12.777+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:12.777339+0000)
, 2022-04-03T03:25:12.879+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:12.879624+0000)

2022-04-03T14:13:10.517 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:23.790+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:25:23.794+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:25:23.881+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:23.882109+0000)

2022-04-03T14:13:10.518 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:34.801+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:25:34.883+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:34.884430+0000)
, 2022-04-03T03:25:34.883+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:34.884625+0000)

2022-04-03T14:13:10.518 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:45.837+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:25:45.886+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:45.886842+0000)
, 2022-04-03T03:25:45.886+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:45.886881+0000)

2022-04-03T14:13:10.519 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:56.888+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:56.889207+0000)
, 2022-04-03T03:25:56.888+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:56.889214+0000)
, 2022-04-03T03:25:57.203+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:57.203605+0000)

2022-04-03T14:13:10.520 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:07.890+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:07.891718+0000)
, 2022-04-03T03:26:08.134+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:08.208+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:08.209611+0000)

2022-04-03T14:13:10.520 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:18.893+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:18.894218+0000)
, 2022-04-03T03:26:18.893+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:18.894202+0000)
, 2022-04-03T03:26:19.211+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.521 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:29.921+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:30.138+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:30.221+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:30.221955+0000)

2022-04-03T14:13:10.522 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:41.039+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:41.228+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:41.228+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:41.228633+0000)

2022-04-03T14:13:10.522 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:52.164+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:52.235+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:52.236703+0000)
, 2022-04-03T03:26:52.235+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:52.236703+0000)

2022-04-03T14:13:10.523 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:03.244+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:03.245150+0000)
, 2022-04-03T03:27:03.244+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:03.245150+0000)
, 2022-04-03T03:27:03.458+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.524 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:14.191+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:27:14.253+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:14.253506+0000)
, 2022-04-03T03:27:14.253+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:14.253506+0000)

2022-04-03T14:13:10.524 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:25.261+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:25.262034+0000)
, 2022-04-03T03:27:25.475+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:27:25.833+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.525 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:36.418+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:27:36.856+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:27:36.856+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:36.857570+0000)

2022-04-03T14:13:10.526 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:47.418+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:27:47.604+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:27:47.862+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:47.863581+0000)

2022-04-03T14:13:10.527 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:58.852+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:27:58.863+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:27:58.869+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:58.869555+0000)

2022-04-03T14:13:10.527 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:09.874+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:09.875422+0000)
, 2022-04-03T03:28:09.892+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:28:09.918+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:09.919854+0000)

2022-04-03T14:13:10.528 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:20.778+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:28:20.880+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:20.881438+0000)
, 2022-04-03T03:28:20.921+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:20.922475+0000)

2022-04-03T14:13:10.529 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:31.886+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:31.887683+0000)
, 2022-04-03T03:28:31.923+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:31.925051+0000)
, 2022-04-03T03:28:31.923+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:31.925157+0000)

2022-04-03T14:13:10.529 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:42.893+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:42.893737+0000)
, 2022-04-03T03:28:42.927+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:42.927393+0000)
, 2022-04-03T03:28:42.927+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:42.927460+0000)

2022-04-03T14:13:10.530 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:53.929+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:53.929969+0000)
, 2022-04-03T03:28:53.975+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:28:54.021+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.531 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:05.002+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:29:05.014+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:29:05.324+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:05.324823+0000)

2022-04-03T14:13:10.531 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:15.935+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:15.935347+0000)
, 2022-04-03T03:29:15.978+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:29:16.014+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.532 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:27.017+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:29:27.333+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:29:27.338+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:27.338553+0000)

2022-04-03T14:13:10.533 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:38.222+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:29:38.344+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:38.345414+0000)
, 2022-04-03T03:29:38.344+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:38.345408+0000)

2022-04-03T14:13:10.533 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:48.942+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:48.942958+0000)
, 2022-04-03T03:29:49.249+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:29:49.351+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:49.352480+0000)

2022-04-03T14:13:10.534 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:00.108+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:30:00.359+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:00.360187+0000)
, 2022-04-03T03:30:00.359+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:00.360187+0000)

2022-04-03T14:13:10.535 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:11.191+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:30:11.366+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:11.367514+0000)
, 2022-04-03T03:30:11.366+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:11.367540+0000)

2022-04-03T14:13:10.535 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:22.373+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:22.374439+0000)
, 2022-04-03T03:30:22.373+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:22.374545+0000)
, 2022-04-03T03:30:22.717+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.536 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:33.381+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:33.381933+0000)
, 2022-04-03T03:30:33.693+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:30:33.852+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.537 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:44.387+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:44.388039+0000)
, 2022-04-03T03:30:44.387+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:44.388046+0000)
, 2022-04-03T03:30:44.679+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.537 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:55.842+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:30:55.864+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:30:55.894+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.538 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:06.881+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:31:06.893+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:31:06.923+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.539 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:17.847+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:31:17.877+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:31:17.918+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.540 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:28.915+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:31:28.951+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:31:28.965+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:28.965161+0000)

2022-04-03T14:13:10.540 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:39.966+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:39.967703+0000)
, 2022-04-03T03:31:39.966+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:39.967823+0000)
, 2022-04-03T03:31:39.972+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.541 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:50.799+0000 7fe007a1a700 -1 osd.7 73 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:31:50.881+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:31:50.969+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:50.969991+0000)

2022-04-03T14:13:10.542 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:01.811+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:32:01.971+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:01.972241+0000)
, 2022-04-03T03:32:01.972+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:01.972381+0000)

2022-04-03T14:13:10.542 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:12.782+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:32:12.973+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:12.974976+0000)
, 2022-04-03T03:32:12.973+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:12.975014+0000)

2022-04-03T14:13:10.543 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:23.976+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:23.977752+0000)
, 2022-04-03T03:32:23.976+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:23.977758+0000)
, 2022-04-03T03:32:24.013+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:24.014393+0000)

2022-04-03T14:13:10.544 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:34.980+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:34.980565+0000)
, 2022-04-03T03:32:35.019+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:35.020504+0000)
, 2022-04-03T03:32:35.455+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:35.456929+0000)

2022-04-03T14:13:10.544 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:45.983+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:45.983282+0000)
, 2022-04-03T03:32:45.983+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:45.983291+0000)
, 2022-04-03T03:32:46.026+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:46.026430+0000)

2022-04-03T14:13:10.545 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:57.031+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:57.032651+0000)
, 2022-04-03T03:32:57.470+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:57.471010+0000)
, 2022-04-03T03:32:57.470+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:57.471010+0000)

2022-04-03T14:13:10.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:08.442+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:33:08.477+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:08.477726+0000)
, 2022-04-03T03:33:08.477+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:08.477930+0000)

2022-04-03T14:13:10.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:19.044+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:19.044597+0000)
, 2022-04-03T03:33:19.483+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:19.484131+0000)
, 2022-04-03T03:33:19.483+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:19.484163+0000)

2022-04-03T14:13:10.547 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:30.490+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:30.490908+0000)
, 2022-04-03T03:33:30.490+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:30.490974+0000)
, 2022-04-03T03:33:30.605+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.548 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:40.996+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:40.997478+0000)
, 2022-04-03T03:33:40.996+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:40.997489+0000)
, 2022-04-03T03:33:41.056+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:41.056929+0000)

2022-04-03T14:13:10.548 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:51.830+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T03:33:51.998+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:52.000141+0000)
, 2022-04-03T03:33:51.998+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:52.000144+0000)

2022-04-03T14:13:10.549 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:03.002+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:03.002679+0000)
, 2022-04-03T03:34:03.002+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:03.002817+0000)
, 2022-04-03T03:34:03.068+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:03.068871+0000)

2022-04-03T14:13:10.550 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:13.871+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:34:14.003+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:14.005074+0000)
, 2022-04-03T03:34:14.003+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:14.005233+0000)

2022-04-03T14:13:10.550 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:25.006+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:25.007551+0000)
, 2022-04-03T03:34:25.080+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:25.081642+0000)
, 2022-04-03T03:34:25.505+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:36.087+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:36.087736+0000)
, 2022-04-03T03:34:36.479+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:34:36.532+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:36.532630+0000)

2022-04-03T14:13:10.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:47.012+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:47.012627+0000)
, 2022-04-03T03:34:47.093+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:47.093969+0000)
, 2022-04-03T03:34:47.466+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:58.509+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:34:58.544+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:58.545593+0000)
, 2022-04-03T03:34:58.544+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:58.545753+0000)

2022-04-03T14:13:10.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:09.550+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:09.552179+0000)
, 2022-04-03T03:35:09.629+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T03:35:09.683+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.554 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:20.543+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:35:20.552+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T03:35:20.558+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:20.559011+0000)

2022-04-03T14:13:10.555 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:31.565+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:31.566261+0000)
, 2022-04-03T03:35:31.565+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:31.566263+0000)
, 2022-04-03T03:35:31.663+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.555 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:42.463+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.444:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e57)
, 2022-04-03T03:35:42.571+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:42.573024+0000)
, 2022-04-03T03:35:42.571+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:42.573058+0000)

2022-04-03T14:13:10.556 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:53.579+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:53.579852+0000)
, 2022-04-03T03:35:53.579+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:53.580138+0000)
, 2022-04-03T03:35:53.595+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.557 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:04.586+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:04.586609+0000)
, 2022-04-03T03:36:04.586+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:04.586819+0000)
, 2022-04-03T03:36:05.030+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:05.031063+0000)

2022-04-03T14:13:10.557 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:15.444+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:36:15.592+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:15.593638+0000)
, 2022-04-03T03:36:15.592+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:15.593679+0000)

2022-04-03T14:13:10.558 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:26.599+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:26.600989+0000)
, 2022-04-03T03:36:27.035+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:27.036511+0000)
, 2022-04-03T03:36:27.035+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:27.036514+0000)

2022-04-03T14:13:10.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:38.039+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:38.039632+0000)
, 2022-04-03T03:36:38.039+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:38.039638+0000)
, 2022-04-03T03:36:38.133+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:48.613+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:48.614122+0000)
, 2022-04-03T03:36:49.042+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:49.042512+0000)
, 2022-04-03T03:36:49.042+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:49.042519+0000)

2022-04-03T14:13:10.561 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:00.044+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:00.045355+0000)
, 2022-04-03T03:37:00.044+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:00.045360+0000)
, 2022-04-03T03:37:00.168+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:00.169355+0000)

2022-04-03T14:13:10.561 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:11.085+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:37:11.125+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:37:11.157+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.562 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:22.050+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:22.051368+0000)
, 2022-04-03T03:37:22.059+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:37:22.087+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.563 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:33.053+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:33.054012+0000)
, 2022-04-03T03:37:33.189+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:37:33.189+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:33.189599+0000)

2022-04-03T14:13:10.563 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:44.055+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:44.056775+0000)
, 2022-04-03T03:37:44.055+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:44.056776+0000)
, 2022-04-03T03:37:44.117+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.564 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:55.211+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:37:55.323+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:37:55.413+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:06.335+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:38:06.343+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:38:06.490+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:17.212+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:17.212423+0000)
, 2022-04-03T03:38:17.297+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:38:17.452+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.566 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:28.385+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:38:28.528+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:38:28.675+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:28.676065+0000)

2022-04-03T14:13:10.567 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:39.682+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:39.682620+0000)
, 2022-04-03T03:38:39.682+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:39.682813+0000)
, 2022-04-03T03:38:39.717+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.567 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:50.607+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:38:50.689+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:50.689720+0000)
, 2022-04-03T03:38:50.689+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:50.689849+0000)

2022-04-03T14:13:10.568 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:01.695+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:01.696708+0000)
, 2022-04-03T03:39:01.716+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:39:01.812+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.569 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:12.806+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:39:12.920+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:12.990+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.569 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:23.783+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:39:23.815+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:24.020+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.570 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:34.833+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:34.963+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:35.080+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.571 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:45.721+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:45.722439+0000)
, 2022-04-03T03:39:45.879+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:46.049+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:56.978+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:39:57.088+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:57.089791+0000)
, 2022-04-03T03:39:57.088+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:57.089818+0000)

2022-04-03T14:13:10.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:08.092+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:08.092513+0000)
, 2022-04-03T03:40:08.092+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:08.092517+0000)
, 2022-04-03T03:40:08.202+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.573 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:19.030+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:40:19.095+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:19.095460+0000)
, 2022-04-03T03:40:19.095+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:19.095459+0000)

2022-04-03T14:13:10.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:30.096+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:30.098204+0000)
, 2022-04-03T03:40:30.096+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:30.098206+0000)
, 2022-04-03T03:40:30.151+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:41.176+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:40:41.290+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:41.291493+0000)
, 2022-04-03T03:40:41.433+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.575 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:52.103+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:52.103673+0000)
, 2022-04-03T03:40:52.262+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:40:52.296+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:52.296984+0000)

2022-04-03T14:13:10.576 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:03.301+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:03.302621+0000)
, 2022-04-03T03:41:03.366+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:41:03.502+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.576 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:14.107+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:14.108685+0000)
, 2022-04-03T03:41:14.202+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:41:14.307+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:14.308714+0000)

2022-04-03T14:13:10.577 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:25.314+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:25.314519+0000)
, 2022-04-03T03:41:25.527+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:41:25.678+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:36.622+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:41:36.714+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:41:36.797+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:36.797653+0000)

2022-04-03T14:13:10.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:47.325+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:47.326405+0000)
, 2022-04-03T03:41:47.480+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:41:47.804+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:47.805209+0000)

2022-04-03T14:13:10.579 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:58.808+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:41:58.810+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:58.811692+0000)
, 2022-04-03T03:41:58.810+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:58.811858+0000)

2022-04-03T14:13:10.580 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:09.818+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:09.818427+0000)
, 2022-04-03T03:42:09.818+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:09.818615+0000)
, 2022-04-03T03:42:10.045+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:20.505+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:42:20.825+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:20.825263+0000)
, 2022-04-03T03:42:20.825+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:20.825430+0000)

2022-04-03T14:13:10.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:31.831+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:31.832525+0000)
, 2022-04-03T03:42:31.831+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:31.832482+0000)
, 2022-04-03T03:42:31.981+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.582 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:43.024+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:42:43.128+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:43.128805+0000)
, 2022-04-03T03:42:43.128+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:43.128963+0000)

2022-04-03T14:13:10.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:53.846+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:53.846758+0000)
, 2022-04-03T03:42:54.036+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:42:54.130+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:54.131165+0000)

2022-04-03T14:13:10.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:05.131+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:05.133312+0000)
, 2022-04-03T03:43:05.132+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:05.133415+0000)
, 2022-04-03T03:43:05.188+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.584 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:15.860+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:15.861059+0000)
, 2022-04-03T03:43:16.134+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:16.135782+0000)
, 2022-04-03T03:43:16.134+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:16.135782+0000)

2022-04-03T14:13:10.585 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:27.137+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:27.138366+0000)
, 2022-04-03T03:43:27.319+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:43:27.381+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:27.382089+0000)

2022-04-03T14:13:10.585 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:38.388+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:38.388300+0000)
, 2022-04-03T03:43:38.403+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:43:38.498+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.586 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:49.142+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:49.143693+0000)
, 2022-04-03T03:43:49.317+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:43:49.393+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:49.394208+0000)

2022-04-03T14:13:10.587 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:00.394+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:00.399+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:00.400656+0000)
, 2022-04-03T03:44:00.549+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.587 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:11.603+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:44:11.895+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:11.896246+0000)
, 2022-04-03T03:44:11.896+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:11.896411+0000)

2022-04-03T14:13:10.588 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:22.413+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:22.413391+0000)
, 2022-04-03T03:44:22.572+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:44:22.882+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.589 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:33.608+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:33.870+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:33.884+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.589 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:44.580+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:44:44.661+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:44.738+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.590 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:55.856+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:55.917+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:55.922+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:55.922691+0000)

2022-04-03T14:13:10.591 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:06.920+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:06.928+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:06.929903+0000)
, 2022-04-03T03:45:06.928+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:06.930027+0000)

2022-04-03T14:13:10.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:17.867+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:17.918+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:17.936+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:17.937463+0000)

2022-04-03T14:13:10.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:28.943+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:28.944139+0000)
, 2022-04-03T03:45:28.943+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:28.944148+0000)
, 2022-04-03T03:45:28.955+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.593 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:39.995+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:40.171+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:40.171878+0000)
, 2022-04-03T03:45:40.171+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:40.171901+0000)

2022-04-03T14:13:10.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:50.983+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:51.121+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:51.173+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:51.174642+0000)

2022-04-03T14:13:10.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:02.010+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:46:02.176+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:02.177446+0000)
, 2022-04-03T03:46:02.176+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:02.177454+0000)

2022-04-03T14:13:10.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:13.179+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:13.180278+0000)
, 2022-04-03T03:46:13.475+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:13.475867+0000)
, 2022-04-03T03:46:13.489+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:24.182+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:24.183096+0000)
, 2022-04-03T03:46:24.202+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:46:24.389+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:35.201+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:46:35.379+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:46:35.414+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.597 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:46.188+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:46.188708+0000)
, 2022-04-03T03:46:46.359+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:46:46.369+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:57.305+0000 7fe007a1a700 -1 osd.7 74 get_health_metrics reporting 19 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:46:57.490+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:46:57.499+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:57.500074+0000)

2022-04-03T14:13:10.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:08.505+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:08.506373+0000)
, 2022-04-03T03:47:08.505+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:47:08.748+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:19.469+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:47:19.511+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:19.512455+0000)
, 2022-04-03T03:47:19.613+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.600 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:30.517+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:30.518426+0000)
, 2022-04-03T03:47:30.572+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:47:30.702+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.600 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:41.656+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:47:42.030+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:42.030438+0000)
, 2022-04-03T03:47:42.030+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:42.030611+0000)

2022-04-03T14:13:10.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:52.621+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:47:52.695+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:47:53.036+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:53.037088+0000)

2022-04-03T14:13:10.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:03.761+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:48:04.043+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:04.044611+0000)
, 2022-04-03T03:48:04.043+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:04.044621+0000)

2022-04-03T14:13:10.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:14.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:48:14.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:48:15.052+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:15.053149+0000)

2022-04-03T14:13:10.603 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:26.061+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:26.062038+0000)
, 2022-04-03T03:48:26.061+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:26.062039+0000)
, 2022-04-03T03:48:26.213+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:26.213715+0000)

2022-04-03T14:13:10.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:36.687+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:48:36.791+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:48:36.814+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:47.711+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:48:47.736+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:48:47.743+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.605 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:58.827+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:48:58.890+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:48:58.926+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:10.069+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:49:10.096+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:10.097831+0000)
, 2022-04-03T03:49:10.096+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:10.097830+0000)

2022-04-03T14:13:10.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:20.910+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:49:21.105+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:21.105929+0000)
, 2022-04-03T03:49:21.105+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:21.105929+0000)

2022-04-03T14:13:10.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:32.113+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:32.115062+0000)
, 2022-04-03T03:49:32.113+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:32.115062+0000)
, 2022-04-03T03:49:32.219+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:43.129+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:49:43.230+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:43.230793+0000)
, 2022-04-03T03:49:43.230+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:43.230899+0000)

2022-04-03T14:13:10.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:54.130+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:54.132137+0000)
, 2022-04-03T03:49:54.210+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:49:54.232+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:54.233525+0000)

2022-04-03T14:13:10.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:05.235+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:05.236384+0000)
, 2022-04-03T03:50:05.235+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:05.236392+0000)
, 2022-04-03T03:50:05.253+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:16.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:50:16.238+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:16.238930+0000)
, 2022-04-03T03:50:16.238+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:16.238930+0000)

2022-04-03T14:13:10.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:27.241+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:27.241855+0000)
, 2022-04-03T03:50:27.412+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:50:27.439+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:38.408+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:50:38.481+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:50:38.623+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:38.624757+0000)

2022-04-03T14:13:10.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:49.246+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:49.247382+0000)
, 2022-04-03T03:50:49.395+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:50:49.407+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:00.418+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T03:51:00.455+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:51:00.636+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:00.636437+0000)

2022-04-03T14:13:10.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:11.642+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:11.642397+0000)
, 2022-04-03T03:51:11.764+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:51:11.879+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:22.403+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:51:22.642+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:51:22.647+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:22.648214+0000)

2022-04-03T14:13:10.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:33.652+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:33.653260+0000)
, 2022-04-03T03:51:33.688+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:51:33.954+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:44.658+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:44.659168+0000)
, 2022-04-03T03:51:44.900+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:51:45.220+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:45.220659+0000)

2022-04-03T14:13:10.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:55.664+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:55.665588+0000)
, 2022-04-03T03:51:55.770+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:51:56.225+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:56.226783+0000)

2022-04-03T14:13:10.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:06.829+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:52:07.232+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:07.233253+0000)
, 2022-04-03T03:52:07.237+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:07.238657+0000)

2022-04-03T14:13:10.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:17.901+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:52:17.903+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:52:18.239+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:18.239938+0000)

2022-04-03T14:13:10.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:28.941+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:52:29.246+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:29.246827+0000)
, 2022-04-03T03:52:29.251+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:29.251741+0000)

2022-04-03T14:13:10.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:40.257+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:40.257971+0000)
, 2022-04-03T03:52:40.271+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:40.272685+0000)
, 2022-04-03T03:52:40.271+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:40.272692+0000)

2022-04-03T14:13:10.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:51.258+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:51.260294+0000)
, 2022-04-03T03:52:51.263+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:51.264386+0000)
, 2022-04-03T03:52:51.274+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:51.275390+0000)

2022-04-03T14:13:10.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:02.269+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:02.270175+0000)
, 2022-04-03T03:53:02.277+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:02.277961+0000)
, 2022-04-03T03:53:02.277+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:02.278000+0000)

2022-04-03T14:13:10.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:13.280+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:13.280447+0000)
, 2022-04-03T03:53:13.358+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:53:13.499+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:24.281+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:24.282504+0000)
, 2022-04-03T03:53:24.281+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:24.282707+0000)
, 2022-04-03T03:53:24.389+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:35.287+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:35.288558+0000)
, 2022-04-03T03:53:35.537+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:53:35.666+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:46.708+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:53:46.728+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:46.729188+0000)
, 2022-04-03T03:53:46.787+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:57.598+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:53:57.620+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:53:57.734+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:57.735442+0000)

2022-04-03T14:13:10.624 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:08.722+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:54:08.740+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:08.741583+0000)
, 2022-04-03T03:54:08.937+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.625 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:19.589+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:54:19.631+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:54:19.747+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:19.747675+0000)

2022-04-03T14:13:10.625 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:30.753+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:30.753746+0000)
, 2022-04-03T03:54:30.837+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:54:30.978+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:42.116+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:54:42.170+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:54:42.298+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:42.300199+0000)

2022-04-03T14:13:10.627 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:52.798+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:54:53.038+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:54:53.296+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:04.085+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:55:04.304+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:04.304935+0000)
, 2022-04-03T03:55:04.304+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:04.305081+0000)

2022-04-03T14:13:10.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:15.307+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:15.307747+0000)
, 2022-04-03T03:55:15.307+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:15.307757+0000)
, 2022-04-03T03:55:15.342+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:15.342552+0000)

2022-04-03T14:13:10.629 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:26.309+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:26.310418+0000)
, 2022-04-03T03:55:26.309+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:26.310463+0000)
, 2022-04-03T03:55:26.347+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:26.348570+0000)

2022-04-03T14:13:10.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:37.312+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:37.312916+0000)
, 2022-04-03T03:55:37.345+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:55:37.354+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:37.355356+0000)

2022-04-03T14:13:10.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:47.630+0000 7f1f38d03700 -1 received  signal: Hangup from killall ceph-osd -1  (PID: 52879) UID: 0
, 2022-04-03T03:55:47.735+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:55:47.797+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:47.798269+0000)

2022-04-03T14:13:10.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:58.801+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:58.802422+0000)
, 2022-04-03T03:55:58.951+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:55:59.241+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.632 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:10.164+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:56:10.317+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:10.318488+0000)
, 2022-04-03T03:56:10.317+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:10.318552+0000)

2022-04-03T14:13:10.632 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:21.319+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:21.320320+0000)
, 2022-04-03T03:56:21.319+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:21.320431+0000)
, 2022-04-03T03:56:21.374+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:21.374940+0000)

2022-04-03T14:13:10.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:32.321+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:32.322217+0000)
, 2022-04-03T03:56:32.379+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:32.379380+0000)
, 2022-04-03T03:56:32.381+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:32.381646+0000)

2022-04-03T14:13:10.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:43.374+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:56:43.382+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:43.383618+0000)
, 2022-04-03T03:56:43.384+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:43.385800+0000)

2022-04-03T14:13:10.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:54.386+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:54.387833+0000)
, 2022-04-03T03:56:54.388+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:54.389889+0000)
, 2022-04-03T03:56:54.645+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.639 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:05.446+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T03:57:05.724+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:57:05.828+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:05.830178+0000)

2022-04-03T14:13:10.640 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:16.720+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:57:16.835+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:16.835493+0000)
, 2022-04-03T03:57:17.179+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.640 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:28.187+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:57:28.217+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:57:28.331+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:28.332086+0000)

2022-04-03T14:13:10.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:39.221+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:57:39.334+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:39.334744+0000)
, 2022-04-03T03:57:39.334+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:39.334761+0000)

2022-04-03T14:13:10.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:50.118+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:57:50.267+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:57:50.336+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:50.337051+0000)

2022-04-03T14:13:10.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:01.326+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:58:01.338+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:01.339454+0000)
, 2022-04-03T03:58:01.338+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:01.339542+0000)

2022-04-03T14:13:10.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:12.336+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:58:12.340+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:12.341985+0000)
, 2022-04-03T03:58:12.340+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:12.342014+0000)

2022-04-03T14:13:10.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:23.343+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:23.344490+0000)
, 2022-04-03T03:58:23.371+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:58:23.434+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:23.435471+0000)

2022-04-03T14:13:10.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:34.346+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:34.347165+0000)
, 2022-04-03T03:58:34.441+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:34.442011+0000)
, 2022-04-03T03:58:34.444+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:34.444557+0000)

2022-04-03T14:13:10.645 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:45.348+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:45.349682+0000)
, 2022-04-03T03:58:45.348+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:45.349730+0000)
, 2022-04-03T03:58:45.447+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:45.448700+0000)

2022-04-03T14:13:10.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:56.454+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:56.454831+0000)
, 2022-04-03T03:58:56.456+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:56.457533+0000)
, 2022-04-03T03:58:56.505+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:07.460+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:07.461148+0000)
, 2022-04-03T03:59:07.462+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:07.463225+0000)
, 2022-04-03T03:59:07.782+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:18.467+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:18.467488+0000)
, 2022-04-03T03:59:18.469+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:18.469304+0000)
, 2022-04-03T03:59:18.512+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:29.474+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:29.475437+0000)
, 2022-04-03T03:59:29.560+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:59:29.848+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:40.695+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:59:40.811+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:59:40.918+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:40.919768+0000)

2022-04-03T14:13:10.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:51.488+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:51.488376+0000)
, 2022-04-03T03:59:51.518+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T03:59:51.702+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.650 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:02.598+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:00:02.795+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:00:02.931+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:02.932822+0000)

2022-04-03T14:13:10.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:13.541+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:00:13.857+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:00:13.938+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:13.939488+0000)

2022-04-03T14:13:10.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:24.801+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:00:24.944+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:24.945659+0000)
, 2022-04-03T04:00:25.330+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:35.951+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:35.952228+0000)
, 2022-04-03T04:00:36.280+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:00:36.374+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:36.374826+0000)

2022-04-03T14:13:10.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:46.902+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:00:46.957+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:46.958722+0000)
, 2022-04-03T04:00:47.307+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:58.298+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:00:58.379+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:58.379779+0000)
, 2022-04-03T04:00:58.379+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:58.379821+0000)

2022-04-03T14:13:10.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:09.380+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:09.382146+0000)
, 2022-04-03T04:01:09.380+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:09.382183+0000)
, 2022-04-03T04:01:09.528+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.655 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:20.124+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:01:20.295+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:01:20.384+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:20.384553+0000)

2022-04-03T14:13:10.655 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:31.234+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:01:31.351+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:01:31.385+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:31.386986+0000)

2022-04-03T14:13:10.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:41.719+0000 7f7975857700 -1 received  signal: Hangup from killall ceph-osd -1  (PID: 53095) UID: 0
, 2022-04-03T04:01:41.989+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:41.990666+0000)
, 2022-04-03T04:01:42.144+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:52.994+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:52.995111+0000)
, 2022-04-03T04:01:53.139+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:01:53.151+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:04.057+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:02:04.237+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:02:04.240+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.658 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:15.217+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:02:15.282+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:02:15.292+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:26.384+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:02:26.395+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:26.396374+0000)
, 2022-04-03T04:02:26.395+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:26.396382+0000)

2022-04-03T14:13:10.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:37.472+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:02:37.475+0000 7fe007a1a700 -1 osd.7 75 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:02:37.589+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:37.590302+0000)

2022-04-03T14:13:10.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:48.593+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:48.594532+0000)
, 2022-04-03T04:02:48.594+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:48.595380+0000)
, 2022-04-03T04:02:48.614+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.661 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:59.597+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:59.598704+0000)
, 2022-04-03T04:02:59.598+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:59.599336+0000)
, 2022-04-03T04:03:00.020+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:00.021414+0000)

2022-04-03T14:13:10.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:10.994+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:03:11.026+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:11.026639+0000)
, 2022-04-03T04:03:11.400+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:22.032+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:22.032707+0000)
, 2022-04-03T04:03:22.406+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:22.406939+0000)
, 2022-04-03T04:03:22.406+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:22.406939+0000)

2022-04-03T14:13:10.663 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:33.408+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:33.409422+0000)
, 2022-04-03T04:03:33.408+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:33.409428+0000)
, 2022-04-03T04:03:33.476+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:43.787+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:03:43.880+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:03:43.953+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:54.806+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:03:54.872+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:03:55.051+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:55.051590+0000)

2022-04-03T14:13:10.665 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:05.664+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:04:05.771+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:04:05.818+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.666 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:16.692+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:04:16.746+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:04:17.064+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:17.064960+0000)

2022-04-03T14:13:10.666 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:28.070+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:28.071762+0000)
, 2022-04-03T04:04:28.307+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:04:28.422+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:28.423138+0000)

2022-04-03T14:13:10.667 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:38.855+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:04:39.077+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:39.077624+0000)
, 2022-04-03T04:04:39.302+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.668 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:50.083+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:50.084108+0000)
, 2022-04-03T04:04:50.291+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:04:50.427+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:50.428040+0000)

2022-04-03T14:13:10.668 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:01.398+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:05:01.430+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:01.430976+0000)
, 2022-04-03T04:05:01.430+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:01.430976+0000)

2022-04-03T14:13:10.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:12.348+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:05:12.433+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:12.433667+0000)
, 2022-04-03T04:05:12.433+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:12.433668+0000)

2022-04-03T14:13:10.670 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:23.435+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:23.436269+0000)
, 2022-04-03T04:05:23.435+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:23.436262+0000)
, 2022-04-03T04:05:23.700+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:23.701146+0000)

2022-04-03T14:13:10.670 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:34.437+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:34.438903+0000)
, 2022-04-03T04:05:34.688+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:05:34.709+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:34.710702+0000)

2022-04-03T14:13:10.671 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:45.439+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:45.441259+0000)
, 2022-04-03T04:05:45.440+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:45.441354+0000)
, 2022-04-03T04:05:45.571+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:56.615+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:05:56.727+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:56.728553+0000)
, 2022-04-03T04:05:56.727+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:56.728571+0000)

2022-04-03T14:13:10.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:07.445+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:07.446177+0000)
, 2022-04-03T04:06:07.718+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:06:07.736+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:07.737248+0000)

2022-04-03T14:13:10.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:18.730+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:06:18.744+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:18.746222+0000)
, 2022-04-03T04:06:18.744+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:18.746222+0000)

2022-04-03T14:13:10.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:29.754+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:29.755616+0000)
, 2022-04-03T04:06:29.825+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:06:29.934+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:40.764+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:40.765141+0000)
, 2022-04-03T04:06:40.881+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:06:40.945+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:51.884+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:06:51.949+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:06:52.006+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.676 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:02.970+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:07:02.988+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:07:03.130+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.676 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:13.866+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:07:13.984+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:07:14.075+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:25.087+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:07:25.175+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:25.175975+0000)
, 2022-04-03T04:07:25.223+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:36.052+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:07:36.181+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:36.182148+0000)
, 2022-04-03T04:07:36.188+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:47.092+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:07:47.187+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:47.188251+0000)
, 2022-04-03T04:07:47.272+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.679 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:58.276+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:07:58.470+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:58.471095+0000)
, 2022-04-03T04:07:58.470+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:58.471267+0000)

2022-04-03T14:13:10.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:09.199+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:09.200016+0000)
, 2022-04-03T04:08:09.211+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:08:09.473+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:09.473722+0000)

2022-04-03T14:13:10.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:20.205+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:20.205915+0000)
, 2022-04-03T04:08:20.476+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:20.476374+0000)
, 2022-04-03T04:08:20.476+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:20.476399+0000)

2022-04-03T14:13:10.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:31.477+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:31.478721+0000)
, 2022-04-03T04:08:31.477+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:31.478817+0000)
, 2022-04-03T04:08:31.692+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.682 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:42.480+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:42.481232+0000)
, 2022-04-03T04:08:42.480+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:42.481278+0000)
, 2022-04-03T04:08:42.628+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:53.483+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:53.483681+0000)
, 2022-04-03T04:08:53.770+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:08:53.797+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:04.673+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:09:04.716+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:09:04.771+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:15.487+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:15.488845+0000)
, 2022-04-03T04:09:15.766+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:09:15.821+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:26.815+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:09:26.888+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:09:26.897+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.685 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:37.652+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:09:37.696+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:09:37.824+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:48.648+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:09:48.854+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:09:48.858+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:59.811+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:09:59.935+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:59.936664+0000)
, 2022-04-03T04:09:59.935+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:59.936663+0000)

2022-04-03T14:13:10.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:10.926+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:10:10.944+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:10.945207+0000)
, 2022-04-03T04:10:10.944+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:10.945409+0000)

2022-04-03T14:13:10.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:21.844+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:10:21.952+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:21.952584+0000)
, 2022-04-03T04:10:21.952+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:21.952609+0000)

2022-04-03T14:13:10.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:32.958+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:32.959902+0000)
, 2022-04-03T04:10:33.086+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:10:33.289+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:33.290994+0000)

2022-04-03T14:13:10.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:43.507+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:43.508391+0000)
, 2022-04-03T04:10:43.698+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:10:43.958+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:54.835+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:10:54.864+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:10:54.969+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:54.970929+0000)

2022-04-03T14:13:10.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:05.888+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:11:05.976+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:05.976366+0000)
, 2022-04-03T04:11:05.976+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:05.976540+0000)

2022-04-03T14:13:10.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:16.980+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:16.981681+0000)
, 2022-04-03T04:11:17.202+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:11:17.309+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:17.310464+0000)

2022-04-03T14:13:10.692 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:28.313+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:28.314853+0000)
, 2022-04-03T04:11:28.502+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:11:28.514+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:28.516044+0000)

2022-04-03T14:13:10.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:39.516+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:39.518039+0000)
, 2022-04-03T04:11:39.516+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:39.518046+0000)
, 2022-04-03T04:11:39.540+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:50.519+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:50.519901+0000)
, 2022-04-03T04:11:50.519+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:50.519910+0000)
, 2022-04-03T04:11:50.599+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.694 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:01.571+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:12:01.600+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:12:01.789+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.695 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:12.769+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:12:12.924+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:12:13.013+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:13.013824+0000)

2022-04-03T14:13:10.695 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:23.956+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:12:24.020+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:24.021293+0000)
, 2022-04-03T04:12:24.020+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:24.021319+0000)

2022-04-03T14:13:10.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:34.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T04:12:34.901+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:12:35.028+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:35.028710+0000)

2022-04-03T14:13:10.697 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:45.955+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:12:46.035+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:46.036425+0000)
, 2022-04-03T04:12:46.035+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:46.036451+0000)

2022-04-03T14:13:10.697 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:57.068+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:12:57.277+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:12:57.359+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:57.360371+0000)

2022-04-03T14:13:10.698 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:08.050+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:08.050416+0000)
, 2022-04-03T04:13:08.065+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:13:08.344+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.699 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:19.057+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:19.057381+0000)
, 2022-04-03T04:13:19.371+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:19.371976+0000)
, 2022-04-03T04:13:19.378+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.699 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:30.373+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:13:30.377+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:30.378673+0000)
, 2022-04-03T04:13:30.540+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:30.541651+0000)

2022-04-03T14:13:10.700 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:41.383+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:41.383730+0000)
, 2022-04-03T04:13:41.434+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:13:41.543+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:41.544052+0000)

2022-04-03T14:13:10.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:52.514+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:13:52.546+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:52.546425+0000)
, 2022-04-03T04:13:52.546+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:52.546657+0000)

2022-04-03T14:13:10.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:03.547+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:03.548680+0000)
, 2022-04-03T04:14:03.547+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:03.548830+0000)
, 2022-04-03T04:14:03.761+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.702 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:14.401+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:14.402687+0000)
, 2022-04-03T04:14:14.549+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:14.551130+0000)
, 2022-04-03T04:14:14.549+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:14.551140+0000)

2022-04-03T14:13:10.703 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:25.553+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:25.553830+0000)
, 2022-04-03T04:14:25.783+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:14:25.901+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.703 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:36.556+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:36.556298+0000)
, 2022-04-03T04:14:36.556+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:36.556317+0000)
, 2022-04-03T04:14:36.655+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.704 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:47.557+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:47.558928+0000)
, 2022-04-03T04:14:47.560+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:14:47.855+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.705 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:58.901+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:14:58.959+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:14:59.008+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.705 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:09.615+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:15:09.756+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:15:09.929+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.706 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:20.808+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:15:21.044+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:15:21.141+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:21.141800+0000)

2022-04-03T14:13:10.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:32.147+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:32.148433+0000)
, 2022-04-03T04:15:32.147+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:32.148964+0000)
, 2022-04-03T04:15:32.191+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:43.155+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:43.155756+0000)
, 2022-04-03T04:15:43.155+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:43.155884+0000)
, 2022-04-03T04:15:43.204+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.708 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:54.162+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:54.163140+0000)
, 2022-04-03T04:15:54.191+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:15:54.460+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:54.460663+0000)

2022-04-03T14:13:10.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:05.374+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:16:05.467+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:05.467230+0000)
, 2022-04-03T04:16:05.525+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:16.259+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:16:16.356+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:16:16.472+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:16.473705+0000)

2022-04-03T14:13:10.710 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:27.479+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:27.480404+0000)
, 2022-04-03T04:16:27.482+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:16:27.581+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:27.581693+0000)

2022-04-03T14:13:10.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:38.456+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:16:38.486+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:38.486941+0000)
, 2022-04-03T04:16:38.584+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:38.584401+0000)

2022-04-03T14:13:10.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:49.492+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:49.493091+0000)
, 2022-04-03T04:16:49.586+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:49.587214+0000)
, 2022-04-03T04:16:49.586+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:49.587216+0000)

2022-04-03T14:13:10.712 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:00.532+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:00.578+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:00.589+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:00.589923+0000)

2022-04-03T14:13:10.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:11.505+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:11.506413+0000)
, 2022-04-03T04:17:11.584+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:11.592+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:11.592655+0000)

2022-04-03T14:13:10.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:22.594+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:22.595250+0000)
, 2022-04-03T04:17:22.595+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:22.595323+0000)
, 2022-04-03T04:17:22.687+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.714 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:33.599+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:33.678+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:34.231+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:34.232218+0000)

2022-04-03T14:13:10.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:44.599+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:44.600201+0000)
, 2022-04-03T04:17:44.701+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:44.791+0000 7fe007a1a700 -1 osd.7 76 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:55.813+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:17:56.246+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:56.247441+0000)
, 2022-04-03T04:17:56.246+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:56.247448+0000)

2022-04-03T14:13:10.716 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:06.882+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:18:06.957+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:18:07.254+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:07.255054+0000)

2022-04-03T14:13:10.717 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:18.027+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:18:18.262+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:18.263103+0000)
, 2022-04-03T04:18:18.262+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:18.263085+0000)

2022-04-03T14:13:10.717 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:29.269+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:29.271134+0000)
, 2022-04-03T04:18:29.272+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:29.273528+0000)
, 2022-04-03T04:18:29.552+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:29.552927+0000)

2022-04-03T14:13:10.718 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:39.785+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:18:39.875+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:18:39.992+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.719 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:50.885+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:18:50.931+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:18:51.065+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.719 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:01.904+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:19:01.941+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:19:02.292+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:02.293170+0000)

2022-04-03T14:13:10.720 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:13.013+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:19:13.032+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:19:13.299+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:13.299813+0000)

2022-04-03T14:13:10.721 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:23.900+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:19:24.306+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:24.306503+0000)
, 2022-04-03T04:19:24.308+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:24.308977+0000)

2022-04-03T14:13:10.721 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:35.312+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:35.312800+0000)
, 2022-04-03T04:19:35.315+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:35.315467+0000)
, 2022-04-03T04:19:35.573+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.722 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:46.123+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:19:46.318+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:46.319573+0000)
, 2022-04-03T04:19:46.321+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:46.322025+0000)

2022-04-03T14:13:10.723 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:57.328+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:57.328706+0000)
, 2022-04-03T04:19:57.603+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:57.605154+0000)
, 2022-04-03T04:19:57.629+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:57.630762+0000)

2022-04-03T14:13:10.723 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:08.332+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:08.332993+0000)
, 2022-04-03T04:20:08.334+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:08.335241+0000)
, 2022-04-03T04:20:08.611+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:08.612226+0000)

2022-04-03T14:13:10.724 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:19.341+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:19.342500+0000)
, 2022-04-03T04:20:19.618+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:19.619189+0000)
, 2022-04-03T04:20:19.635+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:19.636193+0000)

2022-04-03T14:13:10.725 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:30.624+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:30.625763+0000)
, 2022-04-03T04:20:30.638+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:30.638963+0000)
, 2022-04-03T04:20:30.638+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:30.638994+0000)

2022-04-03T14:13:10.726 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:41.618+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:20:41.631+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:41.632236+0000)
, 2022-04-03T04:20:41.640+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:41.641291+0000)

2022-04-03T14:13:10.726 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:52.643+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:52.643618+0000)
, 2022-04-03T04:20:52.643+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:52.643840+0000)
, 2022-04-03T04:20:52.686+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.727 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:03.658+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:21:03.758+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:21:04.026+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.728 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:14.647+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:14.648566+0000)
, 2022-04-03T04:21:14.648+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:14.650118+0000)
, 2022-04-03T04:21:14.873+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.728 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:25.818+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:21:26.082+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:21:26.099+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.729 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:36.959+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:21:37.276+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:21:37.386+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:37.386663+0000)

2022-04-03T14:13:10.730 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:47.911+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:21:47.944+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:21:48.344+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.731 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:59.028+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:21:59.398+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:59.399992+0000)
, 2022-04-03T04:21:59.402+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:59.404196+0000)

2022-04-03T14:13:10.731 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:09.979+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:22:10.406+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:10.406484+0000)
, 2022-04-03T04:22:10.410+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:10.411210+0000)

2022-04-03T14:13:10.732 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:21.412+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:21.413421+0000)
, 2022-04-03T04:22:21.416+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:21.417852+0000)
, 2022-04-03T04:22:21.503+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.733 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:32.420+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:32.420808+0000)
, 2022-04-03T04:22:32.424+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:32.424913+0000)
, 2022-04-03T04:22:32.549+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.733 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:43.431+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:43.431763+0000)
, 2022-04-03T04:22:43.468+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:22:43.506+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.734 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:54.438+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:54.438568+0000)
, 2022-04-03T04:22:54.473+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:22:54.672+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:54.672366+0000)

2022-04-03T14:13:10.735 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:05.506+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:23:05.674+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:05.675211+0000)
, 2022-04-03T04:23:05.674+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:05.675212+0000)

2022-04-03T14:13:10.735 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:16.467+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:23:16.600+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:23:16.677+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:16.678086+0000)

2022-04-03T14:13:10.736 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:27.680+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:27.680994+0000)
, 2022-04-03T04:23:27.680+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:27.680993+0000)
, 2022-04-03T04:23:27.730+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:27.731489+0000)

2022-04-03T14:13:10.737 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:38.683+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:38.683875+0000)
, 2022-04-03T04:23:38.683+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:38.683883+0000)
, 2022-04-03T04:23:38.700+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.737 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:49.685+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:49.686552+0000)
, 2022-04-03T04:23:49.685+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:49.686576+0000)
, 2022-04-03T04:23:49.744+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:49.744973+0000)

2022-04-03T14:13:10.738 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:00.744+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:24:00.751+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:00.752020+0000)
, 2022-04-03T04:24:01.093+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:11.758+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:11.758689+0000)
, 2022-04-03T04:24:11.850+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:24:12.045+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:22.881+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:24:23.057+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:24:23.131+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.740 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:34.085+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:24:34.195+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:24:34.418+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:44.945+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:24:45.034+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:24:45.206+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:56.251+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:24:56.340+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:24:56.509+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:56.510176+0000)

2022-04-03T14:13:10.742 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:07.497+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:07.515+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:07.517191+0000)
, 2022-04-03T04:25:07.517+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:07.518630+0000)

2022-04-03T14:13:10.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:18.275+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:18.462+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:18.523+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:18.524545+0000)

2022-04-03T14:13:10.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:29.531+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:29.532313+0000)
, 2022-04-03T04:25:29.531+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:29.532355+0000)
, 2022-04-03T04:25:29.631+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.744 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:40.541+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:40.541302+0000)
, 2022-04-03T04:25:40.541+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:40.541300+0000)
, 2022-04-03T04:25:40.712+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:40.712454+0000)

2022-04-03T14:13:10.745 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:51.549+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:51.550416+0000)
, 2022-04-03T04:25:51.586+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:51.713+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:51.714786+0000)

2022-04-03T14:13:10.745 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:02.627+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:26:02.716+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:02.717548+0000)
, 2022-04-03T04:26:02.716+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:02.717559+0000)

2022-04-03T14:13:10.746 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:13.699+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:26:13.719+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:13.720011+0000)
, 2022-04-03T04:26:13.719+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:13.720018+0000)

2022-04-03T14:13:10.747 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:24.722+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:24.722621+0000)
, 2022-04-03T04:26:24.722+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:24.722636+0000)
, 2022-04-03T04:26:24.838+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:24.838250+0000)

2022-04-03T14:13:10.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:35.724+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:35.725128+0000)
, 2022-04-03T04:26:35.775+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:26:35.843+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:35.845090+0000)

2022-04-03T14:13:10.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:46.726+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:46.727782+0000)
, 2022-04-03T04:26:46.726+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:46.727787+0000)
, 2022-04-03T04:26:46.816+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.749 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:57.858+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:57.859144+0000)
, 2022-04-03T04:26:57.888+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:26:58.043+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:08.865+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:08.866044+0000)
, 2022-04-03T04:27:09.034+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:27:09.079+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:19.828+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:27:19.872+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:19.872894+0000)
, 2022-04-03T04:27:20.135+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.751 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:30.879+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:30.879576+0000)
, 2022-04-03T04:27:31.044+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:27:31.125+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.752 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:41.886+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:41.886530+0000)
, 2022-04-03T04:27:41.986+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:27:42.139+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:52.975+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:27:53.314+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:27:53.543+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:04.234+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:28:04.645+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:04.645880+0000)
, 2022-04-03T04:28:04.645+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:04.646076+0000)

2022-04-03T14:13:10.754 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:15.071+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:28:15.393+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:28:15.653+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:15.653523+0000)

2022-04-03T14:13:10.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:26.660+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:26.661097+0000)
, 2022-04-03T04:28:26.660+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:26.661101+0000)
, 2022-04-03T04:28:26.753+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:26.753859+0000)

2022-04-03T14:13:10.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:37.668+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:37.669041+0000)
, 2022-04-03T04:28:37.755+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:37.756357+0000)
, 2022-04-03T04:28:37.755+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:37.756384+0000)

2022-04-03T14:13:10.756 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:48.676+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:48.677112+0000)
, 2022-04-03T04:28:48.676+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:48.677120+0000)
, 2022-04-03T04:28:48.758+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:48.758822+0000)

2022-04-03T14:13:10.757 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:59.684+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:59.685066+0000)
, 2022-04-03T04:28:59.761+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:59.761419+0000)
, 2022-04-03T04:28:59.761+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:59.761446+0000)

2022-04-03T14:13:10.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:10.763+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:10.763954+0000)
, 2022-04-03T04:29:10.763+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:10.764050+0000)
, 2022-04-03T04:29:10.780+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:21.765+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:21.766498+0000)
, 2022-04-03T04:29:21.765+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:21.766556+0000)
, 2022-04-03T04:29:21.854+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.759 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:32.768+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:32.769103+0000)
, 2022-04-03T04:29:32.885+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:29:32.953+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:32.954294+0000)

2022-04-03T14:13:10.760 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:43.771+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:43.771452+0000)
, 2022-04-03T04:29:43.954+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:29:43.961+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:43.961422+0000)

2022-04-03T14:13:10.760 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:54.966+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:54.967643+0000)
, 2022-04-03T04:29:54.978+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:29:55.198+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.761 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:06.056+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:30:06.277+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:30:06.415+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.762 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:16.979+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:16.980186+0000)
, 2022-04-03T04:30:17.034+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:30:17.308+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.763 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:28.368+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:30:28.404+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:30:28.746+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:28.746923+0000)

2022-04-03T14:13:10.763 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:39.349+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:30:39.753+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:39.754936+0000)
, 2022-04-03T04:30:39.753+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:39.754942+0000)

2022-04-03T14:13:10.764 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:50.214+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:30:50.471+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:30:50.762+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:50.762767+0000)

2022-04-03T14:13:10.765 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:01.456+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:31:01.769+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:01.770130+0000)
, 2022-04-03T04:31:01.769+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:01.770132+0000)

2022-04-03T14:13:10.765 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:12.455+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:31:12.777+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:12.777841+0000)
, 2022-04-03T04:31:12.777+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:12.777945+0000)

2022-04-03T14:13:10.766 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:23.784+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:23.785320+0000)
, 2022-04-03T04:31:23.784+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:23.785396+0000)
, 2022-04-03T04:31:23.793+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:23.794472+0000)

2022-04-03T14:13:10.767 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:34.791+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:34.792712+0000)
, 2022-04-03T04:31:34.796+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:34.797160+0000)
, 2022-04-03T04:31:34.796+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:34.797165+0000)

2022-04-03T14:13:10.767 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:45.799+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:45.799673+0000)
, 2022-04-03T04:31:45.799+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:45.799695+0000)
, 2022-04-03T04:31:45.799+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:45.800038+0000)

2022-04-03T14:13:10.768 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:56.807+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:56.807335+0000)
, 2022-04-03T04:31:56.807+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:56.807844+0000)
, 2022-04-03T04:31:56.965+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:07.847+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:32:07.918+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:32:07.995+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:18.820+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:18.821300+0000)
, 2022-04-03T04:32:18.878+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:32:18.970+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.770 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:29.905+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:32:29.945+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:30.061+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:30.061935+0000)

2022-04-03T14:13:10.771 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:40.915+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:40.960+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:32:41.067+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:41.068543+0000)

2022-04-03T14:13:10.771 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:52.046+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:52.074+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:52.075279+0000)
, 2022-04-03T04:32:52.154+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.772 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:03.117+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:33:03.151+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:33:03.460+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.773 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:14.088+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:14.088446+0000)
, 2022-04-03T04:33:14.124+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:33:14.131+0000 7fe007a1a700 -1 osd.7 77 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.773 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:25.225+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:33:25.693+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:33:25.821+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:25.822123+0000)

2022-04-03T14:13:10.774 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:35.874+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:35.875794+0000)
, 2022-04-03T04:33:36.101+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:36.102134+0000)
, 2022-04-03T04:33:36.169+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.775 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:46.883+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:46.883649+0000)
, 2022-04-03T04:33:46.883+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:46.883647+0000)
, 2022-04-03T04:33:47.108+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:47.108903+0000)

2022-04-03T14:13:10.776 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:58.115+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:58.115441+0000)
, 2022-04-03T04:33:58.136+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:33:58.291+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.776 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:09.155+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:34:09.162+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:34:09.762+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.777 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:20.114+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:20.128+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:34:20.129066+0000)
, 2022-04-03T04:34:20.171+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.778 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:31.220+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:34:31.230+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:31.836+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:34:31.837073+0000)

2022-04-03T14:13:10.778 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:42.183+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:42.322+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:34:42.838+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:34:42.839562+0000)

2022-04-03T14:13:10.779 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:53.319+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:53.841+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:34:53.842082+0000)
, 2022-04-03T04:34:53.841+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:34:53.842103+0000)

2022-04-03T14:13:10.780 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:04.843+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:04.844510+0000)
, 2022-04-03T04:35:04.843+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:04.844486+0000)
, 2022-04-03T04:35:04.938+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:04.938688+0000)

2022-04-03T14:13:10.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:15.445+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:35:15.846+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:15.847229+0000)
, 2022-04-03T04:35:15.846+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:15.847308+0000)

2022-04-03T14:13:10.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:26.848+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:26.849594+0000)
, 2022-04-03T04:35:26.848+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:26.849678+0000)
, 2022-04-03T04:35:26.953+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:26.954559+0000)

2022-04-03T14:13:10.782 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:37.851+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:37.852263+0000)
, 2022-04-03T04:35:37.961+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:37.962420+0000)
, 2022-04-03T04:35:37.961+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:37.962462+0000)

2022-04-03T14:13:10.783 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:48.854+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:48.854951+0000)
, 2022-04-03T04:35:48.916+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:35:48.969+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:48.969958+0000)

2022-04-03T14:13:10.783 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:59.871+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:35:59.976+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:59.977156+0000)
, 2022-04-03T04:35:59.976+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:59.977249+0000)

2022-04-03T14:13:10.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:10.984+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:10.985080+0000)
, 2022-04-03T04:36:10.992+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:36:11.196+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:11.197133+0000)

2022-04-03T14:13:10.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:21.879+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:36:21.992+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:21.992795+0000)
, 2022-04-03T04:36:21.992+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:21.992933+0000)

2022-04-03T14:13:10.785 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:33.000+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:33.000831+0000)
, 2022-04-03T04:36:33.000+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:33.000847+0000)
, 2022-04-03T04:36:33.210+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:33.211019+0000)

2022-04-03T14:13:10.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:44.021+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:36:44.023+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:36:44.072+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:10.787 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:55.045+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:36:55.223+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:55.224680+0000)
, 2022-04-03T04:36:55.602+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.787 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:06.231+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:06.231794+0000)
, 2022-04-03T04:37:06.667+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:37:06.810+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:17.030+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:17.030973+0000)
, 2022-04-03T04:37:17.238+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:17.238589+0000)
, 2022-04-03T04:37:17.564+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.789 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:28.661+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:37:28.781+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:37:28.822+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.789 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:39.847+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:37:39.867+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:37:39.879+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:39.880547+0000)

2022-04-03T14:13:10.790 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:50.715+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:37:50.797+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:37:50.881+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:50.883166+0000)

2022-04-03T14:13:10.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:01.781+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:38:01.788+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:38:01.885+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:01.885734+0000)

2022-04-03T14:13:10.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:12.755+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:38:12.780+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:38:12.887+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:12.888265+0000)

2022-04-03T14:13:10.792 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:23.889+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:23.890614+0000)
, 2022-04-03T04:38:23.889+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:23.890655+0000)
, 2022-04-03T04:38:23.899+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:34.892+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:34.893383+0000)
, 2022-04-03T04:38:34.942+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:38:35.098+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:35.098643+0000)

2022-04-03T14:13:10.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:45.895+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:45.895818+0000)
, 2022-04-03T04:38:45.895+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:45.895871+0000)
, 2022-04-03T04:38:46.024+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.794 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:56.951+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:38:57.114+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:57.116217+0000)
, 2022-04-03T04:38:57.114+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:57.116213+0000)

2022-04-03T14:13:10.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:08.126+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:08.126716+0000)
, 2022-04-03T04:39:08.126+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:08.126715+0000)
, 2022-04-03T04:39:08.139+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:18.903+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:18.903394+0000)
, 2022-04-03T04:39:19.136+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:19.136867+0000)
, 2022-04-03T04:39:19.136+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:19.136867+0000)

2022-04-03T14:13:10.796 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:30.098+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:39:30.146+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:30.147255+0000)
, 2022-04-03T04:39:30.146+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:30.147255+0000)

2022-04-03T14:13:10.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:41.156+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:41.157433+0000)
, 2022-04-03T04:39:41.328+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:41.330012+0000)
, 2022-04-03T04:39:41.681+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:52.167+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:52.167874+0000)
, 2022-04-03T04:39:52.167+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:52.167874+0000)
, 2022-04-03T04:39:52.336+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:52.336624+0000)

2022-04-03T14:13:10.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:03.177+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:03.177947+0000)
, 2022-04-03T04:40:03.343+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:03.343370+0000)
, 2022-04-03T04:40:03.641+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:14.186+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:14.188224+0000)
, 2022-04-03T04:40:14.186+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:14.188224+0000)
, 2022-04-03T04:40:14.348+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:14.349777+0000)

2022-04-03T14:13:10.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:25.355+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:25.356492+0000)
, 2022-04-03T04:40:25.754+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:40:25.753+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:36.670+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:40:36.858+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:40:36.906+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:47.369+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:47.369845+0000)
, 2022-04-03T04:40:47.590+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:40:47.798+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.802 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:58.779+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:40:58.883+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:40:58.926+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:58.927239+0000)

2022-04-03T14:13:10.802 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:09.866+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:09.908+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:41:09.929+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:41:09.929921+0000)

2022-04-03T14:13:10.803 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:20.901+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:20.932+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:41:20.932678+0000)
, 2022-04-03T04:41:20.932+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:41:20.932712+0000)

2022-04-03T14:13:10.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:31.935+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:41:31.935619+0000)
, 2022-04-03T04:41:31.963+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:32.037+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:43.052+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:41:43.103+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:41:43.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:10.805 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:53.941+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:54.015+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:41:54.051+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:05.070+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:42:05.241+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:42:05.285+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:05.286159+0000)

2022-04-03T14:13:10.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:16.149+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:42:16.175+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:42:16.266+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.807 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:27.195+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:42:27.303+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:27.304640+0000)
, 2022-04-03T04:42:27.303+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:27.304674+0000)

2022-04-03T14:13:10.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:38.312+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:38.313097+0000)
, 2022-04-03T04:42:38.312+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:38.313097+0000)
, 2022-04-03T04:42:38.436+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:38.436767+0000)

2022-04-03T14:13:10.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:49.070+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:42:49.323+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:49.323607+0000)
, 2022-04-03T04:42:49.323+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:49.323607+0000)

2022-04-03T14:13:10.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:00.336+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:00.337638+0000)
, 2022-04-03T04:43:00.338+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:00.339312+0000)
, 2022-04-03T04:43:00.448+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:00.449818+0000)

2022-04-03T14:13:10.810 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:11.455+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:11.456496+0000)
, 2022-04-03T04:43:11.833+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:43:11.840+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:22.351+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:22.352817+0000)
, 2022-04-03T04:43:22.462+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:22.463261+0000)
, 2022-04-03T04:43:22.892+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:33.470+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:33.470197+0000)
, 2022-04-03T04:43:33.901+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:43:33.965+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:33.966092+0000)

2022-04-03T14:13:10.812 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:44.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T04:43:44.475+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:44.476573+0000)
, 2022-04-03T04:43:44.874+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:55.930+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:43:55.970+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:55.971135+0000)
, 2022-04-03T04:43:55.970+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:55.971280+0000)

2022-04-03T14:13:10.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:06.973+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:06.973821+0000)
, 2022-04-03T04:44:06.973+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:06.973811+0000)
, 2022-04-03T04:44:07.116+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.814 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:17.702+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:17.975+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:17.976398+0000)
, 2022-04-03T04:44:17.975+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:17.976402+0000)

2022-04-03T14:13:10.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:28.977+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:28.978913+0000)
, 2022-04-03T04:44:29.091+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:44:29.149+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:40.170+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:40.188+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:44:40.403+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:40.403773+0000)

2022-04-03T14:13:10.816 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:50.983+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:50.983550+0000)
, 2022-04-03T04:44:51.041+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:51.183+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:02.055+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:45:02.072+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:45:02.418+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:02.419483+0000)

2022-04-03T14:13:10.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:13.426+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:13.427088+0000)
, 2022-04-03T04:45:13.426+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:13.427114+0000)
, 2022-04-03T04:45:13.528+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:13.529545+0000)

2022-04-03T14:13:10.818 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:24.107+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:45:24.434+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:24.435047+0000)
, 2022-04-03T04:45:24.434+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:24.435046+0000)

2022-04-03T14:13:10.819 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:35.442+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:35.442935+0000)
, 2022-04-03T04:45:35.442+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:35.442923+0000)
, 2022-04-03T04:45:35.542+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:35.542975+0000)

2022-04-03T14:13:10.819 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:46.268+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:45:46.449+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:46.450641+0000)
, 2022-04-03T04:45:46.449+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:46.450654+0000)

2022-04-03T14:13:10.820 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:57.457+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:57.458531+0000)
, 2022-04-03T04:45:57.555+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:57.556403+0000)
, 2022-04-03T04:45:57.798+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.821 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:08.563+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:08.563453+0000)
, 2022-04-03T04:46:08.619+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:46:08.904+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.822 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:19.472+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:19.472576+0000)
, 2022-04-03T04:46:19.560+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:46:19.569+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:19.570049+0000)

2022-04-03T14:13:10.822 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:30.562+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:46:30.576+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:30.576955+0000)
, 2022-04-03T04:46:30.892+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:41.837+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:46:42.007+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:42.008120+0000)
, 2022-04-03T04:46:42.007+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:42.008169+0000)

2022-04-03T14:13:10.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:52.810+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:46:52.899+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:46:53.010+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:53.010686+0000)

2022-04-03T14:13:10.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:03.727+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:47:04.011+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:04.013022+0000)
, 2022-04-03T04:47:04.011+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:04.013141+0000)

2022-04-03T14:13:10.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:15.014+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:15.015297+0000)
, 2022-04-03T04:47:15.014+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:15.015517+0000)
, 2022-04-03T04:47:15.027+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.826 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:26.017+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:26.017828+0000)
, 2022-04-03T04:47:26.017+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:26.017861+0000)
, 2022-04-03T04:47:26.113+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.826 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:37.020+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:37.020560+0000)
, 2022-04-03T04:47:37.020+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:37.020560+0000)
, 2022-04-03T04:47:37.335+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.827 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:47.980+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:47:48.022+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:48.023080+0000)
, 2022-04-03T04:47:48.022+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:48.023080+0000)

2022-04-03T14:13:10.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:59.100+0000 7fa0fd7a6700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:47:59.111+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:47:59.122+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:48:10.191+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:48:10.311+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:48:10.563+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:10.563898+0000)

2022-04-03T14:13:10.829 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:48:21.080+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:48:21.131+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:48:21.374+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:48:32.252+0000 7fe007a1a700 -1 osd.7 78 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:48:32.487+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:48:32.584+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:32.584973+0000)

2022-04-03T14:13:10.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:48:43.036+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:43.037152+0000)
, 2022-04-03T04:48:43.036+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:43.037134+0000)
, 2022-04-03T04:48:43.182+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.831 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:48:53.866+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:48:54.039+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:54.040006+0000)
, 2022-04-03T04:48:54.039+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:54.040009+0000)

2022-04-03T14:13:10.832 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:49:05.042+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:49:05.042849+0000)
, 2022-04-03T04:49:05.044+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:49:05.123+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:49:16.033+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:49:16.044+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:49:16.045463+0000)
, 2022-04-03T04:49:16.044+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:49:16.045464+0000)

2022-04-03T14:13:10.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:49:27.091+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:49:27.186+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:49:27.360+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.834 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:49:38.208+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:49:38.228+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:49:38.644+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:49:38.644711+0000)

2022-04-03T14:13:10.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:49:49.198+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:49:49.244+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:49:49.261+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:50:00.181+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:50:00.275+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:50:00.659+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:00.660728+0000)

2022-04-03T14:13:10.836 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:50:11.668+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:11.668535+0000)
, 2022-04-03T04:50:11.668+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:11.668539+0000)
, 2022-04-03T04:50:11.711+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:11.712658+0000)

2022-04-03T14:13:10.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:50:22.227+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:50:22.676+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:22.677105+0000)
, 2022-04-03T04:50:22.676+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:22.677189+0000)

2022-04-03T14:13:10.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:50:33.684+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:33.685400+0000)
, 2022-04-03T04:50:33.684+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:33.685444+0000)
, 2022-04-03T04:50:33.725+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:33.726168+0000)

2022-04-03T14:13:10.838 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:50:44.693+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:44.694395+0000)
, 2022-04-03T04:50:44.732+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:44.732801+0000)
, 2022-04-03T04:50:44.808+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)

2022-04-03T14:13:10.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:50:55.704+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:55.704623+0000)
, 2022-04-03T04:50:55.716+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.300:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e46)
, 2022-04-03T04:50:55.739+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:50:55.739636+0000)

2022-04-03T14:13:10.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:51:06.714+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:06.714673+0000)
, 2022-04-03T04:51:06.745+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:06.746154+0000)
, 2022-04-03T04:51:06.997+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:51:17.724+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:17.725021+0000)
, 2022-04-03T04:51:17.724+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:17.725053+0000)
, 2022-04-03T04:51:17.751+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:17.752691+0000)

2022-04-03T14:13:10.841 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:51:28.759+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:28.759630+0000)
, 2022-04-03T04:51:28.881+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:51:29.077+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:29.078218+0000)

2022-04-03T14:13:10.841 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:51:39.974+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:51:40.080+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:40.081014+0000)
, 2022-04-03T04:51:40.080+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:40.081016+0000)

2022-04-03T14:13:10.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:51:50.772+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:50.773290+0000)
, 2022-04-03T04:51:51.003+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:51:51.082+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:51:51.083453+0000)

2022-04-03T14:13:10.843 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:52:01.975+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:52:02.063+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:52:02.085+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:02.086236+0000)

2022-04-03T14:13:10.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:52:13.088+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:13.089039+0000)
, 2022-04-03T04:52:13.088+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:13.089048+0000)
, 2022-04-03T04:52:13.404+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:52:23.943+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:52:24.091+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:24.091719+0000)
, 2022-04-03T04:52:24.091+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:24.091730+0000)

2022-04-03T14:13:10.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:52:35.093+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:35.094459+0000)
, 2022-04-03T04:52:35.093+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:35.094459+0000)
, 2022-04-03T04:52:35.109+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:52:46.096+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:46.097242+0000)
, 2022-04-03T04:52:46.096+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:46.097263+0000)
, 2022-04-03T04:52:46.582+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:52:57.099+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:57.100139+0000)
, 2022-04-03T04:52:57.099+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:52:57.100123+0000)
, 2022-04-03T04:52:57.566+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:53:08.102+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:08.102941+0000)
, 2022-04-03T04:53:08.589+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:53:08.811+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:08.811970+0000)

2022-04-03T14:13:10.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:53:19.104+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:19.105729+0000)
, 2022-04-03T04:53:19.104+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:19.105733+0000)
, 2022-04-03T04:53:19.536+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:53:30.107+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:30.108338+0000)
, 2022-04-03T04:53:30.416+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:53:30.616+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:53:41.645+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:53:41.842+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:41.842329+0000)
, 2022-04-03T04:53:41.842+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:53:41.842880+0000)

2022-04-03T14:13:10.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:53:52.428+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:53:52.653+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:53:52.673+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:10.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:54:03.640+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:54:03.641+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:54:03.857+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:03.858164+0000)

2022-04-03T14:13:10.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:54:14.746+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:54:14.864+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:14.865441+0000)
, 2022-04-03T04:54:14.868+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:14.868656+0000)

2022-04-03T14:13:10.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:54:25.734+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:54:25.872+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:25.872693+0000)
, 2022-04-03T04:54:25.877+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:25.877678+0000)

2022-04-03T14:13:10.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:54:36.878+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:36.879940+0000)
, 2022-04-03T04:54:36.884+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:36.885928+0000)
, 2022-04-03T04:54:36.884+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:36.885928+0000)

2022-04-03T14:13:10.853 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:54:47.748+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)
, 2022-04-03T04:54:47.885+0000 7faaa15cd700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:47.886744+0000)
, 2022-04-03T04:54:47.895+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:47.895786+0000)

2022-04-03T14:13:10.854 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:54:58.904+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:58.905655+0000)
, 2022-04-03T04:54:58.904+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:54:58.905656+0000)
, 2022-04-03T04:54:58.957+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:55:09.914+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:09.914944+0000)
, 2022-04-03T04:55:09.914+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:09.914953+0000)
, 2022-04-03T04:55:10.131+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:10.131721+0000)

2022-04-03T14:13:10.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:55:20.920+0000 7f796e9ea700 -1 osd.3 42 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.155:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:55:20.921+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:20.923115+0000)
, 2022-04-03T04:55:20.922+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:20.923485+0000)

2022-04-03T14:13:10.856 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:55:31.930+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:31.931008+0000)
, 2022-04-03T04:55:31.930+0000 7f1f25c70700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:31.931048+0000)
, 2022-04-03T04:55:32.135+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:32.136402+0000)

2022-04-03T14:13:10.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:55:43.137+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:43.138840+0000)
, 2022-04-03T04:55:43.137+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:43.138994+0000)
, 2022-04-03T04:55:43.583+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:55:53.946+0000 7f79627c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:53.946909+0000)
, 2022-04-03T04:55:54.140+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:54.141148+0000)
, 2022-04-03T04:55:54.141+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:55:54.141361+0000)

2022-04-03T14:13:10.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:56:05.142+0000 7f89ca8d3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:56:05.143454+0000)
, 2022-04-03T04:56:05.142+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:56:05.143518+0000)
, 2022-04-03T04:56:05.450+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:56:16.144+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:56:16.145842+0000)
, 2022-04-03T04:56:16.591+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:56:16.606+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)

2022-04-03T14:13:10.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:56:27.148+0000 7f29d1744700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:56:27.148325+0000)
, 2022-04-03T04:56:27.532+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:56:27.654+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:10.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:56:38.564+0000 7faaad7f3700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4219.0:12 2.1 2.3e6a8311 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)
, 2022-04-03T04:56:38.675+0000 7fe007a1a700 -1 osd.7 79 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.155:4 3.15 3:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:56:38.918+0000 7f1f31e96700 -1 osd.0 42 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:23332 3.1 3:86ee9bce:::500.00000001:head [write 1818101~7272 in=7272b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e42)

2022-04-03T14:13:12.192 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-11430-gcc9ed509 (cc9ed509562d767c1a5d8fc115593bc1453dfea1) quincy (dev)
,  1: /lib64/libpthread.so.0(+0x12c20) [0x7ff8cd3a9c20]
,  2: gsignal()

2022-04-03T14:13:12.193 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:42:10.117+0000 7f7b589b7700 -1 osd.5 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:42:10.130+0000 7fa9c620e700 -1 osd.4 38 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:42:10.366+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.194 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:42:29.772+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:42:29.897+0000 7fa9c620e700 -1 osd.4 38 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:42:30.175+0000 7f7b589b7700 -1 osd.5 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.195 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:42:50.093+0000 7fa9c620e700 -1 osd.4 38 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:42:50.201+0000 7f7b589b7700 -1 osd.5 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:42:50.313+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.196 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:43:09.899+0000 7fa9c620e700 -1 osd.4 38 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:43:09.937+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 25 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:43:10.116+0000 7f7b589b7700 -1 osd.5 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.196 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:43:29.543+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:43:29.886+0000 7fa9c620e700 -1 osd.4 38 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:43:29.910+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 25 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.197 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:43:49.955+0000 7f7b589b7700 -1 osd.5 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:43:50.015+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 25 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:43:50.297+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.198 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:44:09.752+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:44:09.816+0000 7f7b589b7700 -1 osd.5 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:44:09.959+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 25 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.198 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:44:32.072+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:44:32.565+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:44:32.839+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 8 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.199 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:44:57.568+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:44:57.953+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:44:58.083+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.200 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:45:22.367+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:45:22.554+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:45:22.977+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 8 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.201 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:45:47.198+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:45:47.379+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:45:47.423+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.201 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:46:12.070+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 8 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T22:46:12.248+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:46:12.580+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.202 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:46:36.616+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:46:37.135+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:46:37.252+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 8 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.203 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:47:01.734+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:47:01.910+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:47:02.276+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 8 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.204 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:47:25.255+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.1.9:12284 3.1d 3.9a77e43d (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e39)
, 2022-04-02T22:47:25.686+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:47:25.759+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.204 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:47:45.377+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:47:45.441+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.1.9:12284 3.1d 3.9a77e43d (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e39)
, 2022-04-02T22:47:45.546+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.205 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:48:04.927+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:48:05.292+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:48:05.315+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.1.9:12284 3.1d 3.9a77e43d (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e39)

2022-04-03T14:13:12.206 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:48:25.278+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:48:25.367+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.1.9:12284 3.1d 3.9a77e43d (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e39)
, 2022-04-02T22:48:25.521+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.207 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:48:45.008+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:48:45.279+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:48:45.386+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.207 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:49:04.931+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:49:05.333+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:49:05.364+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.208 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:49:25.409+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:49:25.574+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:49:25.640+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.209 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:49:44.854+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:49:45.580+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:49:45.629+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.210 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:50:04.824+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:50:04.889+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:50:05.480+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:50:25.063+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:50:25.375+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:50:25.500+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:50:44.806+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:50:45.101+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:50:45.373+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.212 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:51:04.784+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:51:04.920+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:51:05.163+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.213 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:51:24.910+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:51:25.226+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:51:25.663+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.213 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:51:44.792+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:51:44.951+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:51:44.981+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.214 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:52:04.724+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:52:04.916+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:52:05.011+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.215 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:52:24.782+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:52:25.017+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:52:25.090+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.216 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:52:44.659+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:52:44.819+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:52:45.060+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.216 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:53:04.307+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:53:04.730+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:53:04.794+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.217 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:53:24.599+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:53:24.861+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:53:25.087+0000 7fa9c620e700 -1 osd.4 39 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.218 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:53:44.357+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:53:44.397+0000 7f7b589b7700 -1 osd.5 39 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:53:44.821+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.218 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:54:00.770+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:54:01.060 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T22:54:01.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.219 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:54:25.116+0000 7fa9c620e700 -1 osd.4 41 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:54:25.152+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:54:25.355+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.220 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:54:49.163+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:54:49.503+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 26 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:54:49.667+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.221 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:55:13.721+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:55:13.920+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:55:14.167+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.221 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-11430-gcc9ed509 (cc9ed509562d767c1a5d8fc115593bc1453dfea1) quincy (dev)
,  1: /lib64/libpthread.so.0(+0x12c20) [0x7f84d43c4c20]
,  2: gsignal()

2022-04-03T14:13:12.222 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:56:07.547+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:56:08.161+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:56:08.297+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 9 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.223 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:56:32.143+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:56:32.184+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 9 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T22:56:32.252+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.224 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:56:57.173+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:56:57.213+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 9 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T22:56:57.420+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.224 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:57:17.285+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:57:17.362+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:57:17.439+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.225 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:57:36.609+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:57:37.211+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:57:37.384+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 9 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.226 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:57:56.595+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:57:57.157+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:57:57.450+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.227 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:58:16.375+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:58:16.642+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:58:16.661+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.227 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:58:35.794+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:58:36.169+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:58:36.218+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.228 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:58:55.817+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:58:56.010+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:58:56.091+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.229 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:59:15.307+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T22:59:15.411+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T22:59:15.768+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.230 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:59:34.904+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:59:34.918+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:59:35.047+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.230 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T22:59:54.875+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:59:54.909+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T22:59:54.909+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.231 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:00:13.992+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:00:14.547+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:00:14.810+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.232 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:00:33.751+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:00:33.926+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:00:33.961+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.232 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:00:53.738+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:00:54.011+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:00:54.108+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.233 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:01:13.277+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:01:13.329+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:01:13.685+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.234 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:01:32.879+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:01:32.937+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:01:32.948+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.235 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:01:52.750+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:01:52.924+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:01:52.939+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.235 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:02:11.950+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:02:12.059+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:02:12.835+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.236 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:02:31.959+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:02:32.017+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:02:32.141+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.237 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:02:51.702+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:02:51.980+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:02:51.983+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.237 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:03:11.061+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:03:11.196+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:03:11.780+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.238 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:03:30.923+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:03:31.032+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:03:31.389+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.239 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:03:50.660+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:03:50.709+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:03:50.894+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:04:10.341+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:04:10.493+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:04:10.561+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:04:30.413+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:04:30.427+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:04:30.504+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.241 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:04:49.473+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:04:49.834+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:04:50.023+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.242 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:05:09.397+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:05:09.444+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:05:09.467+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.243 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:05:29.339+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:05:29.530+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:05:29.538+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.244 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:05:48.686+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:05:49.242+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:05:49.259+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.244 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:06:08.285+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:06:08.536+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:06:08.972+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.245 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:06:28.302+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:06:28.624+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:06:28.831+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.246 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:06:47.821+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:06:48.054+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:06:48.224+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.246 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:07:07.351+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:07:07.678+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:07:07.855+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.247 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:07:27.390+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:07:27.723+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:07:28.084+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.248 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:07:47.066+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:07:47.385+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:07:47.386+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.249 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:08:06.740+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:08:06.865+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:08:07.161+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.249 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:08:26.812+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:08:27.026+0000 7fa9c620e700 -1 osd.4 42 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:08:27.109+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.250 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:08:46.211+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:08:46.385+0000 7f7b589b7700 -1 osd.5 42 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:08:46.527+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.251 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:09:03.873+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:09:04.137+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:09:04.190+0000 7f7b589b7700 -1 osd.5 43 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.252 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:09:23.692+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:09:24.074+0000 7f7b589b7700 -1 osd.5 44 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:09:24.272+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.252 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:09:43.674+0000 7fa9c620e700 -1 osd.4 44 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:09:43.697+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:09:43.957+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.253 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:10:02.024+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:10:02.769+0000 7fa9c620e700 -1 osd.4 44 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:10:02.872+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.254 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:10:22.011+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:10:22.090+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:10:22.091+0000 7f7b589b7700 -1 osd.5 44 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.255 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:10:41.392+0000 7fa9c620e700 -1 osd.4 44 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:10:42.006+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 28 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:10:42.030+0000 7f7b589b7700 -1 osd.5 44 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.255 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:11:01.437+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:11:01.624+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:11:01.777+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.256 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:11:21.133+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T23:11:21.165+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:11:21.516+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.257 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:11:40.778+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:11:40.818+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:11:41.045+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.257 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:00.661+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:00.793+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:00.860+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.258 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:17.838+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:17.931+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:18.084+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.259 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:34.420+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:34.439 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:12:34.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.260 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:12:50.888+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:50.937+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:12:51.015+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.260 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:07.123+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T23:13:07.372+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:13:07.412+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.261 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:23.928+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:13:23.944+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:13:24.105+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.262 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:39.996+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:13:40.092+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T23:13:40.426+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.263 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:13:56.768+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:13:56.870+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 11 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T23:13:56.974+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.263 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:14:13.085+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:14:13.249+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:14:13.493+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.264 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:14:29.914+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:14:30.002+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:14:30.080+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.265 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:14:46.079+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:14:46.191+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:14:46.456+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.266 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:02.858+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:15:02.983+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:15:03.015+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.266 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:19.114+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:15:19.164+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:15:19.416+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.267 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:35.423+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:15:35.740+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:15:35.765+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.268 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:15:52.155+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:15:52.455+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:15:52.477+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.269 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:08.551+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:16:08.553+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:16:08.947+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.269 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:25.175+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:16:25.584+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:16:25.662+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.270 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:41.708+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:16:41.750+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:16:41.768+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.271 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:16:58.107+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:16:58.468+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:16:58.625+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.272 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:17:14.613+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:17:14.727+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:17:14.774+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.272 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:17:31.135+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:17:31.376+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:17:31.555+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.273 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:17:47.545+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:17:47.613+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:17:47.723+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.274 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:18:04.337+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:18:04.436+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:18:04.475+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.275 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:18:20.475+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:18:20.504+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:18:20.782+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.275 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:18:36.903+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:18:36.951+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:18:37.361+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.276 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:18:53.596+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:18:53.647+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:18:53.900+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.277 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:19:09.862+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:19:09.947+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:19:10.289+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.278 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:19:26.503+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:19:26.737+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:19:26.779+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.278 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:19:42.638+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:19:43.058+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:19:43.069+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.279 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:19:59.423+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:19:59.628+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:19:59.961+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.280 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:20:15.701+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:20:15.803+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:20:15.829+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.281 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:20:32.324+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:20:32.554+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:20:32.628+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.281 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:20:48.526+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:20:48.664+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:20:48.894+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.282 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:21:05.152+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:21:05.556+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:21:05.603+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.283 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:21:21.598+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:21:21.682+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:21:21.993+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.284 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:21:37.975+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:21:38.071+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:21:38.326+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.285 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:21:54.776+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:21:54.937+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:21:55.078+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.286 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:22:10.990+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:22:11.183+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:22:11.474+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.286 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:22:27.923+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:22:27.923+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:22:28.036+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.287 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:22:44.163+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:22:44.211+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:22:44.401+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.288 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:23:00.774+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:23:00.942+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:23:01.050+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.289 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:23:17.115+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:23:17.194+0000 7fa9c620e700 -1 osd.4 45 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:23:17.209+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.289 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:23:33.703+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:23:33.987+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:23:34.208+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.290 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:23:50.108+0000 7f7b589b7700 -1 osd.5 45 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:23:50.198+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:23:50.208+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.291 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:24:06.241+0000 7fe9577ef700 -1 MDSIOContextBase: failed with -110, restarting...
, 2022-04-02T23:24:06.241+0000 7fe9587f1700 -1 mds.1.purge_queue operator(): Error -110 loading Journaler
, 2022-04-02T23:24:06.241+0000 7fe956fee700 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -110 ((110) Connection timed out)

2022-04-03T14:13:12.291 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:24:21.129+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:24:21.197+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:24:21.236+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.292 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:24:37.211+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:24:37.213+0000 7fa9c620e700 -1 osd.4 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:24:37.268+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.293 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:24:54.051+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:24:54.138+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:24:54.246+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.294 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:10.215+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:25:10.326+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:25:10.353+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.294 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:26.939+0000 7fa9c620e700 -1 osd.4 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:25:27.141+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:25:27.281+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.295 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:43.284+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:25:43.336+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:25:43.360+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.296 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:25:59.900+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:25:59.906+0000 7fa9c620e700 -1 osd.4 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:26:00.303+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.297 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:26:15.490+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:26:15.510+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:26:15.725+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.297 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:26:31.962+0000 7fa9c620e700 -1 osd.4 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:26:32.058+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:26:32.434+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.298 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:26:48.373+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:26:48.555+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:26:48.643+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.299 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:04.807+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 30 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:27:05.228+0000 7fa9c620e700 -1 osd.4 47 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:27:05.429+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.300 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:21.369+0000 7f7b589b7700 -1 osd.5 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:27:21.399+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:27:21.706+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.301 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:37.789+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:27:37.812+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:27:38.195+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 13 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)

2022-04-03T14:13:12.301 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:27:54.349+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 13 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T23:27:54.389+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:27:54.882+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.302 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:28:10.731+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:28:10.858+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:28:11.252+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.303 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:28:27.380+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:28:27.524+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:28:27.662+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.304 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:28:43.536+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:28:44.074+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:28:44.165+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.304 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:00.423+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:29:00.487+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:29:00.524+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.305 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:16.639+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:29:16.669+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:29:17.127+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.306 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:33.325+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:29:33.383+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 13 slow ops, oldest is osd_repop(mds.2.11:6011 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'220, mlcod=38'216)
, 2022-04-02T23:29:33.514+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:29:49.528+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:29:49.837+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:29:50.249+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:30:06.241+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:30:06.456+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:30:06.809+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.308 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:30:22.840+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:30:22.842+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:30:23.213+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.309 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:30:39.014+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:30:39.081+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:30:39.251+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.310 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:30:55.750+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:30:55.758+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:30:55.916+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.310 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:31:11.954+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:31:11.963+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:31:12.201+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.311 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:31:28.627+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:31:28.766+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:31:28.794+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:31:44.919+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:31:45.049+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:31:45.091+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:32:01.550+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:32:01.629+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:32:01.694+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.313 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:32:17.811+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:32:18.049+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:32:18.060+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:32:34.700+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:32:34.796+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:32:34.859+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:32:51.029+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:32:51.176+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:32:51.331+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:33:07.703+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:33:07.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:33:07.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.316 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:33:24.037+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:33:24.162+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:33:24.192+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:33:40.335+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:33:40.380+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:33:40.850+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:33:57.160+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:33:57.203+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:33:57.466+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.318 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:34:13.519+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:34:13.538+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:34:14.039+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.319 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:34:30.017+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:34:30.231+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:34:30.544+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.320 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:34:46.397+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:34:46.562+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:34:46.614+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.320 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:35:03.012+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:35:03.198+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:35:03.384+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.321 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:35:19.252+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:35:19.620+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:35:19.685+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.322 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:35:36.128+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:35:36.222+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:35:36.243+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.323 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:35:52.309+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:35:52.583+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:35:52.642+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.323 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:36:08.683+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:36:08.904+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:36:09.082+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.324 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:36:25.487+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:36:25.495+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:36:25.711+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.325 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:36:41.633+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:36:41.997+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:36:42.088+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.325 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:36:58.457+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:36:58.471+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:36:58.549+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.326 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:37:14.570+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:37:15.057+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:37:15.152+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.327 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:37:31.335+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:37:31.338+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:37:31.592+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.328 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:37:47.521+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:37:47.652+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:37:47.901+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.328 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:38:04.172+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:38:04.356+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:38:04.818+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.329 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:38:20.779+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:38:20.927+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:38:20.950+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.330 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:38:37.177+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:38:37.297+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:38:37.844+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:38:53.886+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:38:53.888+0000 7fa9c620e700 -1 osd.4 48 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:38:53.968+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:09.941+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:39:10.003+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:39:10.310+0000 7f7b589b7700 -1 osd.5 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.332 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:25.027+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:39:25.052+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:39:25.343+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.333 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:41.295+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:39:41.509+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:39:41.616+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.333 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:39:57.884+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:39:58.146+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:39:58.158+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:40:14.147+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:40:14.600+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:40:14.609+0000 7fa9c620e700 -1 osd.4 50 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:40:30.727+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:40:30.893+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:40:30.921+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:40:46.829+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:40:47.163+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:40:47.707+0000 7fa9c620e700 -1 osd.4 50 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:41:03.694+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:41:03.695+0000 7fa9c620e700 -1 osd.4 50 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:41:03.723+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.337 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:41:19.814+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:41:20.309+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:41:20.530+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:41:36.584+0000 7fa9c620e700 -1 osd.4 50 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:41:36.718+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 32 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:41:36.896+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:41:52.896+0000 7f7b589b7700 -1 osd.5 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:41:53.231+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:41:53.420+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:09.591+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:42:09.607+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:42:09.619+0000 7fa9c620e700 -1 osd.4 50 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.340 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:25.526+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:42:25.568+0000 7fa9c620e700 -1 osd.4 51 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:42:25.580+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:41.754+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:42:41.811+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:42:41.930+0000 7f7b589b7700 -1 osd.5 51 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:42:58.669+0000 7fa9c620e700 -1 osd.4 51 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:42:58.720+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:42:58.902+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.342 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:43:14.907+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:43:15.036+0000 7f7b589b7700 -1 osd.5 51 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:43:15.046+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:43:31.678+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 16 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:43:31.748+0000 7fa9c620e700 -1 osd.4 51 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:43:31.780+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:43:47.917+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:43:47.934+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:43:48.004+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:04.581+0000 7fa9c620e700 -1 osd.4 51 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:44:04.596+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 16 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:44:04.881+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:20.822+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:44:20.890+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:44:21.001+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.346 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:37.321+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 16 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:44:37.602+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:44:37.712+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.346 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:44:53.732+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:44:53.756+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:44:54.098+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:09.778+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:45:09.780+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:45:09.873+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:21.944+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:45:21.945230+0000)
, 2022-04-02T23:45:21.948+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:45:22.008+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:34.271+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:45:34.632+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:45:34.725+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:45:34.726215+0000)

2022-04-03T14:13:12.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:46.729+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:45:46.729952+0000)
, 2022-04-02T23:45:46.838+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:45:46.871+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:45:59.112+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:45:59.274+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:45:59.668+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:11.470+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:46:11.736+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:46:11.737423+0000)
, 2022-04-02T23:46:11.823+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:23.964+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:46:23.965516+0000)
, 2022-04-02T23:46:23.996+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:46:24.010+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:36.360+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:46:36.744+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:46:36.745389+0000)
, 2022-04-02T23:46:36.850+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:46:48.749+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:46:48.749501+0000)
, 2022-04-02T23:46:48.788+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:46:48.888+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.354 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:01.119+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:47:01.232+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:47:01.356+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:13.679+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:47:13.756+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:47:13.756932+0000)
, 2022-04-02T23:47:13.854+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:25.896+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:47:25.985+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:47:25.986004+0000)
, 2022-04-02T23:47:26.243+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:38.495+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:47:38.626+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:47:38.693+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:47:50.768+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:47:50.768486+0000)
, 2022-04-02T23:47:50.797+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:47:50.798+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:03.252+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:48:03.460+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:48:03.515+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:15.610+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-02T23:48:15.620+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:48:15.649+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.359 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:28.005+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:48:28.005862+0000)
, 2022-04-02T23:48:28.320+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:48:28.449+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:40.574 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:48:40.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-02T23:48:40.598+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:48:52.856+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:48:52.905+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:48:53.013+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:48:53.013906+0000)

2022-04-03T14:13:12.361 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:05.459+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:49:05.516+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:49:05.578+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.362 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:17.795+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:49:17.796010+0000)
, 2022-04-02T23:49:17.805+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:49:17.826+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:30.365+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:49:30.514+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:49:30.568+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:42.655+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:49:42.803+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:49:42.803757+0000)
, 2022-04-02T23:49:42.934+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.364 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:49:55.033+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:49:55.034833+0000)
, 2022-04-02T23:49:55.033+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:49:55.495+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.365 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:07.764+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:07.811+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:50:07.812327+0000)
, 2022-04-02T23:50:07.829+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:19.889+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:19.989+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:20.041+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:50:20.042566+0000)

2022-04-03T14:13:12.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:32.198+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:32.369+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:50:32.814+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:44.793+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:44.824+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:50:44.824663+0000)
, 2022-04-02T23:50:45.049+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:50:45.050257+0000)

2022-04-03T14:13:12.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:50:57.244+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:57.313+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:50:57.344+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.370 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:09.971+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:51:10.006+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:10.057+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:10.058400+0000)

2022-04-03T14:13:12.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:22.062+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:51:22.119+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:22.215+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:34.463+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:34.741+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:34.837+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:46.845+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:51:46.845156+0000)
, 2022-04-02T23:51:46.954+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:46.968+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:51:59.188+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:59.334+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:51:59.721+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:11.757+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:52:11.853+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:11.853977+0000)
, 2022-04-02T23:52:12.053+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:24.081+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:24.081903+0000)
, 2022-04-02T23:52:24.212+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:52:24.332+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:36.850+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:52:36.861+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:36.862093+0000)
, 2022-04-02T23:52:37.010+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:52:49.034+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:52:49.088+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:52:49.089790+0000)
, 2022-04-02T23:52:49.188+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:01.650+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:01.767+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:53:01.868+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:01.869738+0000)

2022-04-03T14:13:12.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:13.884+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:13.921+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:53:14.098+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:14.098974+0000)

2022-04-03T14:13:12.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:26.169+0000 7fa9c620e700 -1 osd.4 52 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:26.251+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:26.672+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:39.062+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:39.107+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:53:39.107230+0000)
, 2022-04-02T23:53:39.159+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:53:51.138+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:51.148+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:53:51.277+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:03.569+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:54:03.890+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:03.890803+0000)
, 2022-04-02T23:54:04.022+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:16.046+0000 7f7b589b7700 -1 osd.5 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:54:16.118+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:16.119037+0000)
, 2022-04-02T23:54:16.155+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:27.189+0000 7f7b589b7700 -1 osd.5 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:54:27.249+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:54:27.283+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:40.012+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:54:40.053+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:54:40.125+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:54:40.126708+0000)

2022-04-03T14:13:12.383 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:54:52.135+0000 7f7b589b7700 -1 osd.5 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:54:52.156+0000 7fa9c620e700 -1 osd.4 54 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:54:52.196+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.384 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:04.442+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:55:04.747+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:55:04.911+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:04.911668+0000)

2022-04-03T14:13:12.385 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:16.914+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:16.915568+0000)
, 2022-04-02T23:55:17.137+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:17.138309+0000)
, 2022-04-02T23:55:17.165+0000 7f7b589b7700 -1 osd.5 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:29.427+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:55:29.431+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:55:29.740+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:41.840+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:55:41.922+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:41.923460+0000)
, 2022-04-02T23:55:42.145+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:55:42.146413+0000)

2022-04-03T14:13:12.387 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:55:54.301+0000 7fa9c620e700 -1 osd.4 54 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:55:54.321+0000 7f7b589b7700 -1 osd.5 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:55:54.398+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:06.908+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:56:06.931+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:06.931630+0000)
, 2022-04-02T23:56:07.154+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:07.155339+0000)

2022-04-03T14:13:12.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:19.158+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:19.159103+0000)
, 2022-04-02T23:56:19.216+0000 7fa9c620e700 -1 osd.4 54 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:56:19.392+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.389 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:31.640+0000 7f7b589b7700 -1 osd.5 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:56:31.802+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:56:31.834+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:43.946+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:56:44.166+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:56:44.167448+0000)
, 2022-04-02T23:56:44.484+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.391 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:56:56.555+0000 7fa9c620e700 -1 osd.4 54 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:56:56.574+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:56:56.697+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.391 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:08.991+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:57:09.176+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:09.176747+0000)
, 2022-04-02T23:57:09.375+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 34 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.392 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:20.457+0000 7f7b589b7700 -1 osd.5 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:57:20.461+0000 7fa9c620e700 -1 osd.4 54 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:57:20.584+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:32.960+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:32.961036+0000)
, 2022-04-02T23:57:32.962+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:57:33.075+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 20 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:45.188+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:57:45.189437+0000)
, 2022-04-02T23:57:45.432+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:57:45.442+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.394 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:57:57.596+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:57:57.915+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 20 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:57:57.931+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.395 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-11430-gcc9ed509 (cc9ed509562d767c1a5d8fc115593bc1453dfea1) quincy (dev)
,  1: /lib64/libpthread.so.0(+0x12c20) [0x7fe770da4c20]
,  2: gsignal()

2022-04-03T14:13:12.396 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:28.770+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:58:28.840+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:58:28.845+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 20 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:41.206+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:58:41.207824+0000)
, 2022-04-02T23:58:41.372+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:58:41.556+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:58:53.437+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:58:53.520+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:58:53.571+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:05.926+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 20 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-02T23:59:05.990+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:05.991716+0000)
, 2022-04-02T23:59:06.193+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.399 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:18.030+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:59:18.219+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:18.220078+0000)
, 2022-04-02T23:59:18.488+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:30.678+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:59:30.809+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:59:30.876+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.401 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:42.948+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-02T23:59:43.003+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:43.004578+0000)
, 2022-04-02T23:59:43.227+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T22:59:43.228315+0000)

2022-04-03T14:13:12.402 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-02T23:59:55.555+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:59:55.784+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-02T23:59:55.833+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 20 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:08.036+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:00:08.073+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:00:08.236+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:08.236859+0000)

2022-04-03T14:13:12.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:20.204+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:00:20.240+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:20.240864+0000)
, 2022-04-03T00:00:20.535+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:32.868+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:00:33.020+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:33.021402+0000)
, 2022-04-03T00:00:33.038+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:45.101+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:00:45.162+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:00:45.248+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:45.249492+0000)

2022-04-03T14:13:12.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:00:57.578+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:00:57.708+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:00:58.028+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:00:58.029558+0000)

2022-04-03T14:13:12.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:10.079+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:01:10.148+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:01:10.257+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:10.257947+0000)

2022-04-03T14:13:12.407 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:22.261+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:22.261982+0000)
, 2022-04-03T00:01:22.622+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:01:22.685+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.407 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:34.897+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:01:35.040+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:35.041892+0000)
, 2022-04-03T00:01:35.117+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.408 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:47.045+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:47.045950+0000)
, 2022-04-03T00:01:47.067+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:01:47.269+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:01:47.270496+0000)

2022-04-03T14:13:12.409 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:01:59.838+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:01:59.966+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:02:00.025+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.410 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:12.146+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:02:12.194+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:02:12.278+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:12.278599+0000)

2022-04-03T14:13:12.411 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:24.280+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:24.282068+0000)
, 2022-04-03T00:02:24.616+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:02:25.012+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:37.058+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:02:37.062+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:37.063272+0000)
, 2022-04-03T00:02:37.130+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:02:49.087+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:02:49.222+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:02:49.289+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:02:49.290466+0000)

2022-04-03T14:13:12.413 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:01.803+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:03:01.831+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:03:01.934+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.414 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:14.075+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:14.075774+0000)
, 2022-04-03T00:03:14.243+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:03:14.298+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:14.298676+0000)

2022-04-03T14:13:12.415 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:26.682+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:03:26.789+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:03:26.797+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.416 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:39.169+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:03:39.306+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:39.307468+0000)
, 2022-04-03T00:03:39.351+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.416 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:03:51.310+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:03:51.311373+0000)
, 2022-04-03T00:03:51.650+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:03:51.805+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.417 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:03.984+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:04:04.051+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:04:04.053+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.418 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:16.021+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:04:16.096+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:16.097237+0000)
, 2022-04-03T00:04:16.319+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:16.319552+0000)

2022-04-03T14:13:12.419 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:28.929+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:04:28.946+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:04:28.973+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.419 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:41.317+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:04:41.326+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:04:41.327508+0000)
, 2022-04-03T00:04:41.543+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.420 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:04:53.456+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:04:53.582+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:04:53.903+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.421 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:06.114+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:06.115058+0000)
, 2022-04-03T00:05:06.250+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:05:06.286+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.421 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:18.279+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:05:18.339+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:18.340102+0000)
, 2022-04-03T00:05:18.483+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.422 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:31.122+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:31.123709+0000)
, 2022-04-03T00:05:31.157+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:05:31.194+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.423 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:43.530+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:05:43.554+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:05:43.563+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.424 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:05:55.549+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:05:56.120+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:05:56.131+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:05:56.132033+0000)

2022-04-03T14:13:12.424 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:08.281+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:06:08.356+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:08.356577+0000)
, 2022-04-03T00:06:08.371+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.425 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:20.410+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:06:20.476+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:06:20.962+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.426 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:33.144+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:33.145182+0000)
, 2022-04-03T00:06:33.288+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:06:33.364+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:33.364886+0000)

2022-04-03T14:13:12.427 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:45.367+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:45.368591+0000)
, 2022-04-03T00:06:45.371+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:06:45.439+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.427 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:06:58.050+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:06:58.147+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:06:58.152+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:06:58.153241+0000)

2022-04-03T14:13:12.428 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:10.463+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:07:10.486+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:07:10.524+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.429 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:22.518+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:07:22.689+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:07:23.087+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.429 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:35.384+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:07:35.385475+0000)
, 2022-04-03T00:07:35.392+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:07:35.451+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.430 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:07:47.412+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:07:47.510+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:07:47.569+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.431 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:00.250+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:08:00.265+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:08:00.393+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:00.393708+0000)

2022-04-03T14:13:12.432 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:12.499+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:08:12.548+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:08:12.692+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.433 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:24.658+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:08:25.129+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:08:25.183+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:25.184799+0000)

2022-04-03T14:13:12.434 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:37.406+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:08:37.406706+0000)
, 2022-04-03T00:08:37.503+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:08:37.535+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.436 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:08:49.491+0000 7f7b589b7700 -1 osd.5 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:08:49.633+0000 7fa9c620e700 -1 osd.4 55 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:08:49.645+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.437 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:02.196+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:02.196795+0000)
, 2022-04-03T00:09:02.339+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:09:02.356+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.438 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:14.386+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:09:14.417+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:14.418963+0000)
, 2022-04-03T00:09:14.474+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.439 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:24.203+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:24.204432+0000)
, 2022-04-03T00:09:24.245+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:09:24.320+0000 7f7b589b7700 -1 osd.5 56 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.441 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:35.290+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:09:35.417+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:09:35.425+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:35.426338+0000)

2022-04-03T14:13:12.442 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:46.211+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:46.212224+0000)
, 2022-04-03T00:09:46.337+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:09:46.382+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.443 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:09:57.433+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:09:57.433940+0000)
, 2022-04-03T00:09:57.553+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:09:57.562+0000 7fa9c620e700 -1 osd.4 57 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.445 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:08.620+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:08.621+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:08.636+0000 7fa9c620e700 -1 osd.4 57 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.446 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:19.440+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:19.441796+0000)
, 2022-04-03T00:10:19.707+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:19.738+0000 7fa9c620e700 -1 osd.4 57 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.447 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:30.730+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:30.740+0000 7fa9c620e700 -1 osd.4 57 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:30.789+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.448 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:41.648+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:41.944+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:10:41.967+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:10:41.969085+0000)

2022-04-03T14:13:12.450 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:10:52.596+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:52.676+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:10:52.971+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.451 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:03.666+0000 7f7b589b7700 -1 osd.5 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:11:03.951+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:03.975+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:03.975473+0000)

2022-04-03T14:13:12.452 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:14.978+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:14.978991+0000)
, 2022-04-03T00:11:14.986 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:11:14.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.454 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:25.981+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:25.982593+0000)
, 2022-04-03T00:11:26.132+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:26.137+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.455 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:37.068+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:11:37.099+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:37.192+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.456 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:47.989+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:47.989666+0000)
, 2022-04-03T00:11:48.042+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:11:48.111+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.458 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:11:59.148+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:11:59.220+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:11:59.258+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:11:59.259059+0000)

2022-04-03T14:13:12.459 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:10.198+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:12:10.252+0000 7fa9c620e700 -1 osd.4 57 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:12:10.262+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:10.263333+0000)

2022-04-03T14:13:12.460 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:21.224+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 36 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:12:21.266+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:21.267467+0000)
, 2022-04-03T00:12:21.276+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.462 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:31.677+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:12:32.004+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:32.004610+0000)
, 2022-04-03T00:12:32.110+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.463 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:43.124+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:12:43.205+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:12:43.274+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:43.275076+0000)

2022-04-03T14:13:12.464 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:12:54.011+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:12:54.011660+0000)
, 2022-04-03T00:12:54.084+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:12:54.125+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.466 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:05.090+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:05.095+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:05.167+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.467 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:16.018+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:16.019388+0000)
, 2022-04-03T00:13:16.062+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:16.173+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.468 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:27.110+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:27.206+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:27.290+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:27.290667+0000)

2022-04-03T14:13:12.469 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:38.210+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:38.294+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:38.294831+0000)
, 2022-04-03T00:13:38.319+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.471 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:13:49.029+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:49.030176+0000)
, 2022-04-03T00:13:49.050+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:13:49.298+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:13:49.298477+0000)

2022-04-03T14:13:12.472 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:00.301+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:00.302513+0000)
, 2022-04-03T00:14:00.350+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 24 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:14:00.358+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.473 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:11.305+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:11.306543+0000)
, 2022-04-03T00:14:11.315+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:14:11.520+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:11.522109+0000)

2022-04-03T14:13:12.475 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:22.309+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:22.310333+0000)
, 2022-04-03T00:14:22.319+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:14:22.321+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 24 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.476 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:33.313+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:33.313995+0000)
, 2022-04-03T00:14:33.423+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:14:33.502+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.477 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:44.499+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:14:44.534+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:44.535241+0000)
, 2022-04-03T00:14:45.047+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:45.048255+0000)

2022-04-03T14:13:12.479 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:14:55.495+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:14:55.498+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:14:55.539+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:14:55.539436+0000)

2022-04-03T14:13:12.480 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:06.511+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:15:06.542+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:06.543872+0000)
, 2022-04-03T00:15:07.016+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.481 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:17.547+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:17.547773+0000)
, 2022-04-03T00:15:17.650+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:15:17.800+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.483 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:28.742+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:15:29.061+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:29.062100+0000)
, 2022-04-03T00:15:29.257+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.484 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:40.065+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:40.065852+0000)
, 2022-04-03T00:15:40.147+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:15:40.185+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.485 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:15:50.774+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:15:51.068+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:15:51.069152+0000)
, 2022-04-03T00:15:51.081+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.486 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:02.071+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:02.072756+0000)
, 2022-04-03T00:16:02.181+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:02.182+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.487 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:13.195+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:16:13.215+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:16:13.348+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:13.349125+0000)

2022-04-03T14:13:12.488 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:24.142+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:24.215+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:16:24.339+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.490 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:35.135+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:35.356+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:35.356521+0000)
, 2022-04-03T00:16:35.415+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.490 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:46.087+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:46.087780+0000)
, 2022-04-03T00:16:46.137+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:16:46.360+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:46.360282+0000)

2022-04-03T14:13:12.491 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:16:57.337+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:16:57.363+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:16:57.364379+0000)
, 2022-04-03T00:16:57.499+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.492 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:08.398+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:17:08.492+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:17:08.586+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:08.587050+0000)

2022-04-03T14:13:12.492 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:19.396+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:17:19.371+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:19.372380+0000)
, 2022-04-03T00:17:19.498+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.493 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:30.547+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:17:30.589+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:17:30.594+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:30.595302+0000)

2022-04-03T14:13:12.494 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:41.598+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:41.599193+0000)
, 2022-04-03T00:17:41.640+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:17:41.702+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.495 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:17:52.602+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:17:52.603345+0000)
, 2022-04-03T00:17:52.689+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:17:52.796+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.495 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:03.690+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:18:03.717+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:18:03.725+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.496 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:14.784+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:18:15.036+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:18:15.115+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:15.115943+0000)

2022-04-03T14:13:12.497 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:25.692+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:18:25.806+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:18:26.057+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.498 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:36.752+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:18:37.064+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:18:37.122+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:37.123057+0000)

2022-04-03T14:13:12.498 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:47.684+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:18:47.771+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:18:48.015+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.499 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:18:58.885+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:18:59.129+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:59.130372+0000)
, 2022-04-03T00:18:59.406+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:18:59.407396+0000)

2022-04-03T14:13:12.500 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:10.133+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:10.133913+0000)
, 2022-04-03T00:19:10.410+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:10.410888+0000)
, 2022-04-03T00:19:10.463+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.500 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:21.076+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:19:21.136+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:21.137359+0000)
, 2022-04-03T00:19:21.395+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.501 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:32.141+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:32.141439+0000)
, 2022-04-03T00:19:32.382+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:19:32.395+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.502 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:43.421+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:43.422071+0000)
, 2022-04-03T00:19:43.442+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:19:43.462+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.503 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:19:54.214+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:19:54.222+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:19:54.424+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:19:54.425714+0000)

2022-04-03T14:13:12.504 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:05.278+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:20:05.428+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:05.429752+0000)
, 2022-04-03T00:20:05.451+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.504 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:16.432+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:16.433534+0000)
, 2022-04-03T00:20:16.454 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:20:16.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.505 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:27.437+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:27.437750+0000)
, 2022-04-03T00:20:27.504+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:20:27.660+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:27.660599+0000)

2022-04-03T14:13:12.506 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:38.664+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:38.664821+0000)
, 2022-04-03T00:20:38.682+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:20:38.739+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.507 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:20:49.444+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:49.445647+0000)
, 2022-04-03T00:20:49.646+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:20:49.667+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:20:49.668851+0000)

2022-04-03T14:13:12.507 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:00.666+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:00.671+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:00.673212+0000)
, 2022-04-03T00:21:00.747+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.508 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:11.725+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:11.886+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:11.943+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.509 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:22.680+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:21:22.681117+0000)
, 2022-04-03T00:21:22.774+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:22.881+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.510 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:33.767+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:33.786+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:21:33.837+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.511 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:44.896+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:44.927+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:21:45.052+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.511 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:21:55.862+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:21:55.884+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:21:55.946+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.512 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:06.897+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:22:07.018+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:22:07.029+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.513 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:17.823+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:22:17.896+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:22:17.916+0000 7f7b589b7700 -1 osd.5 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.514 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:29.008+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:22:29.158+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:22:29.200+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:29.200282+0000)

2022-04-03T14:13:12.514 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:40.092+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:22:40.202+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:40.203985+0000)
, 2022-04-03T00:22:40.484+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:40.486040+0000)

2022-04-03T14:13:12.515 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:22:50.945+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:22:51.026+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:22:51.206+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:22:51.207776+0000)

2022-04-03T14:13:12.516 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:02.060+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:23:02.210+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:02.211358+0000)
, 2022-04-03T00:23:02.493+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:02.493789+0000)

2022-04-03T14:13:12.516 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:13.497+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:13.497608+0000)
, 2022-04-03T00:23:13.550+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:23:13.584+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.517 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:24.218+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:24.218766+0000)
, 2022-04-03T00:23:24.488+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:23:24.501+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:24.501479+0000)

2022-04-03T14:13:12.518 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:35.427+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:23:35.505+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:35.505382+0000)
, 2022-04-03T00:23:35.533+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.519 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:46.540+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:23:46.545+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:23:46.684+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.519 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:23:57.508+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:23:57.511+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:57.512604+0000)
, 2022-04-03T00:23:57.735+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:23:57.737022+0000)

2022-04-03T14:13:12.520 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:08.547+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:24:08.594+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:24:08.740+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:08.741073+0000)

2022-04-03T14:13:12.521 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:19.590+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:24:19.598+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:24:19.619+0000 7fa9c620e700 -1 osd.4 58 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.521 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:29.606+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:24:29.655+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:24:29.747+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:29.748158+0000)

2022-04-03T14:13:12.522 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:40.750+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:40.751491+0000)
, 2022-04-03T00:24:40.782+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:24:41.030+0000 7f7b589b7700 -1 osd.5 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.523 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:24:51.754+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:24:51.755125+0000)
, 2022-04-03T00:24:51.762+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:24:51.878+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.524 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:02.758+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:25:02.759098+0000)
, 2022-04-03T00:25:02.963+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:02.989+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.524 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:14.041+0000 7f7b589b7700 -1 osd.5 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:14.071+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:25:14.191+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.525 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:25.023+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:25.043+0000 7f7b589b7700 -1 osd.5 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:25.088+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.526 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:35.985+0000 7fa9c620e700 -1 osd.4 60 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:36.010+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:36.144+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.527 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:47.079+0000 7fa9c620e700 -1 osd.4 60 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:47.197+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:25:47.209+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.527 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:25:58.092+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:58.101+0000 7fa9c620e700 -1 osd.4 60 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:25:58.269+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:25:58.270601+0000)

2022-04-03T14:13:12.528 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:09.097+0000 7fa9c620e700 -1 osd.4 60 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:09.273+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:09.274327+0000)
, 2022-04-03T00:26:09.406+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.529 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:19.948+0000 7f7b589b7700 -1 osd.5 60 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:19.997+0000 7fa9c620e700 -1 osd.4 60 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:20.225+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.530 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:31.018+0000 7fa9c620e700 -1 osd.4 60 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:31.202+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:31.280+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:31.281751+0000)

2022-04-03T14:13:12.530 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:42.288+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:42.533+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:26:42.573+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:42.574379+0000)

2022-04-03T14:13:12.531 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:26:53.167+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:26:53.288+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:26:53.289247+0000)
, 2022-04-03T00:26:53.562+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.532 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:04.292+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:04.293104+0000)
, 2022-04-03T00:27:04.340+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:04.582+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:04.582592+0000)

2022-04-03T14:13:12.533 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:15.586+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:15.586835+0000)
, 2022-04-03T00:27:15.748+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:27:15.768+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.533 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:26.299+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:26.299963+0000)
, 2022-04-03T00:27:26.566+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:27:26.590+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:26.590679+0000)

2022-04-03T14:13:12.534 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:36.915+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:27:37.081+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:27:37.302+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:37.303609+0000)

2022-04-03T14:13:12.535 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:47.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:27:48.050+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:27:48.242+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.536 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:27:59.125+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:27:59.194+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:27:59.310+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:27:59.311033+0000)

2022-04-03T14:13:12.536 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:10.168+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:28:10.314+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:10.315114+0000)
, 2022-04-03T00:28:10.607+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:10.607626+0000)

2022-04-03T14:13:12.537 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:21.196+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:28:21.211+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:28:21.319+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:21.319415+0000)

2022-04-03T14:13:12.538 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:32.290+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:28:32.322+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:32.323335+0000)
, 2022-04-03T00:28:32.421+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.538 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:43.365+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:28:43.619+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:43.619821+0000)
, 2022-04-03T00:28:43.838+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:43.838992+0000)

2022-04-03T14:13:12.539 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:28:54.401+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:28:54.490+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:28:54.623+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:28:54.623968+0000)

2022-04-03T14:13:12.540 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:05.610+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:29:05.626+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:05.627683+0000)
, 2022-04-03T00:29:05.845+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:05.846669+0000)

2022-04-03T14:13:12.540 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:16.631+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:16.631603+0000)
, 2022-04-03T00:29:16.850+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:16.850701+0000)
, 2022-04-03T00:29:17.083+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.541 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:27.634+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:27.635366+0000)
, 2022-04-03T00:29:27.853+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:27.854700+0000)
, 2022-04-03T00:29:28.162+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.542 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:38.857+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:38.858622+0000)
, 2022-04-03T00:29:39.099+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:29:39.173+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.543 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:29:49.642+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:49.643164+0000)
, 2022-04-03T00:29:49.862+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:29:49.862782+0000)
, 2022-04-03T00:29:50.077+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.543 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:00.866+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:00.866799+0000)
, 2022-04-03T00:30:01.024+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:01.102+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.544 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:12.230+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:12.279+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:12.333+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.545 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:23.008+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:23.077+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:23.262+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:34.141+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:34.169+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:34.336+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:45.368+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:45.369065+0000)
, 2022-04-03T00:30:45.464+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:45.469+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.547 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:30:56.040+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:30:56.372+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:30:56.372713+0000)
, 2022-04-03T00:30:56.386+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.548 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:07.376+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:07.376660+0000)
, 2022-04-03T00:31:07.397+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:31:07.404+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.548 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:18.380+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:18.380564+0000)
, 2022-04-03T00:31:18.515+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:31:18.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.549 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:29.513+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:31:29.531+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:31:29.536+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.550 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:40.507+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:31:40.536+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:31:40.578+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:31:51.382+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:31:51.390+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:31:51.391879+0000)
, 2022-04-03T00:31:51.558+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:02.395+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:02.395541+0000)
, 2022-04-03T00:32:02.485+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:32:02.500+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:13.568+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:32:13.696+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:13.697123+0000)
, 2022-04-03T00:32:13.755+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:24.501+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:32:24.597+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:32:24.700+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:24.701116+0000)

2022-04-03T14:13:12.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:35.509+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:32:35.704+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:35.705193+0000)
, 2022-04-03T00:32:35.805+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.554 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:46.708+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:46.709318+0000)
, 2022-04-03T00:32:46.873+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:32:46.927+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:46.928465+0000)

2022-04-03T14:13:12.555 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:32:57.713+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:57.713429+0000)
, 2022-04-03T00:32:57.886+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:32:57.931+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:32:57.932340+0000)

2022-04-03T14:13:12.556 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:08.887+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:33:08.936+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:08.936338+0000)
, 2022-04-03T00:33:08.980+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.556 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:19.721+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:19.722329+0000)
, 2022-04-03T00:33:19.773+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:33:19.939+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:19.940538+0000)

2022-04-03T14:13:12.557 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:30.854+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:33:30.943+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:30.944414+0000)
, 2022-04-03T00:33:31.022+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.558 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:42.034+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:33:42.058+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:33:42.243+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:33:52.952+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:33:52.953238+0000)
, 2022-04-03T00:33:52.953+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:33:53.026+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:03.985+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:34:04.107+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:34:04.187+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.560 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:15.186+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:34:15.427+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:34:15.442+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:15.443157+0000)

2022-04-03T14:13:12.561 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:26.074+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:34:26.158+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:34:26.286+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.561 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:37.165+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:34:37.307+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:34:37.450+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:37.451139+0000)

2022-04-03T14:13:12.562 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:48.373+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:34:48.454+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:48.454889+0000)
, 2022-04-03T00:34:48.686+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.563 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:34:59.458+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:34:59.459026+0000)
, 2022-04-03T00:34:59.504+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:34:59.736+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.564 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:10.582+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:35:10.620+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:35:10.763+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:10.763676+0000)

2022-04-03T14:13:12.564 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:21.465+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:21.466611+0000)
, 2022-04-03T00:35:21.599+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:35:21.735+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:32.563+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:35:32.768+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:35:32.771+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:32.771920+0000)

2022-04-03T14:13:12.566 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:43.689+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:35:43.775+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:43.776102+0000)
, 2022-04-03T00:35:43.882+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.566 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:35:54.685+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:35:54.779+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:35:54.780+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:35:54.780354+0000)

2022-04-03T14:13:12.567 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:05.727+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:36:05.784+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:05.784387+0000)
, 2022-04-03T00:36:05.897+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.568 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:16.787+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:16.788524+0000)
, 2022-04-03T00:36:16.882+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:36:17.005+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:17.006437+0000)

2022-04-03T14:13:12.569 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:27.791+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:27.792954+0000)
, 2022-04-03T00:36:27.831+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:36:27.999+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.569 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:38.796+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:38.796989+0000)
, 2022-04-03T00:36:39.013+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:39.014155+0000)
, 2022-04-03T00:36:39.086+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.570 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:36:49.784+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:36:49.801+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:49.801283+0000)
, 2022-04-03T00:36:50.018+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:36:50.018689+0000)

2022-04-03T14:13:12.571 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:00.805+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T00:37:00.858+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:37:01.022+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:01.022851+0000)

2022-04-03T14:13:12.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:12.025+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:12.026747+0000)
, 2022-04-03T00:37:12.315+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:37:12.505+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:12.506381+0000)

2022-04-03T14:13:12.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:22.911+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:37:23.030+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:23.031312+0000)
, 2022-04-03T00:37:23.385+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.573 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:34.034+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:34.035477+0000)
, 2022-04-03T00:37:34.420+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:37:34.496+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:45.517+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:45.518587+0000)
, 2022-04-03T00:37:45.518+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:37:45.645+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:37:56.444+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:37:56.466+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:37:56.522+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:37:56.522705+0000)

2022-04-03T14:13:12.575 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:07.397+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:38:07.525+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:07.526817+0000)
, 2022-04-03T00:38:07.637+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.576 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:18.529+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:18.530813+0000)
, 2022-04-03T00:38:18.579+0000 7f7b589b7700 -1 osd.5 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:38:18.719+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.577 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:29.533+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:29.534593+0000)
, 2022-04-03T00:38:29.554+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:38:29.744+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.577 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:40.537+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:40.538542+0000)
, 2022-04-03T00:38:40.770+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:38:40.787+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:38:51.517+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:38:51.542+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:38:51.542657+0000)
, 2022-04-03T00:38:51.679+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.579 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:02.545+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:02.546564+0000)
, 2022-04-03T00:39:02.630+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:39:02.849+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:02.850123+0000)

2022-04-03T14:13:12.579 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:13.853+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:13.854335+0000)
, 2022-04-03T00:39:13.935+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:39:14.048+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.580 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:24.725+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:39:24.857+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:24.858620+0000)
, 2022-04-03T00:39:24.945+0000 7fa9c620e700 -1 osd.4 61 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:34.613+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:39:34.862+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:34.862529+0000)
, 2022-04-03T00:39:34.882+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.582 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:45.866+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:45.866667+0000)
, 2022-04-03T00:39:45.992+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:39:46.046+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.582 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:39:56.870+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:39:56.870967+0000)
, 2022-04-03T00:39:56.987+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:39:57.039+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:07.934+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:40:08.045+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:40:08.095+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:08.096576+0000)

2022-04-03T14:13:12.584 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:19.053+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:40:19.099+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:19.100683+0000)
, 2022-04-03T00:40:19.224+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.584 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:30.101+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:40:30.104+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:30.105130+0000)
, 2022-04-03T00:40:30.180+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.585 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:41.109+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:41.109417+0000)
, 2022-04-03T00:40:41.111+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:40:41.374+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.586 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:40:52.034+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:40:52.113+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:40:52.114044+0000)
, 2022-04-03T00:40:52.113+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.587 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:03.117+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:03.118405+0000)
, 2022-04-03T00:41:03.135+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:41:03.434+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.588 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:14.305+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:41:14.591+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:14.592537+0000)
, 2022-04-03T00:41:14.724+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.588 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:25.317+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:41:25.340+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:41:25.595+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:25.596483+0000)

2022-04-03T14:13:12.589 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:36.302+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:41:36.600+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:36.600490+0000)
, 2022-04-03T00:41:36.658+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.590 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:47.604+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:47.604689+0000)
, 2022-04-03T00:41:47.619+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:41:47.913+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:47.914099+0000)

2022-04-03T14:13:12.591 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:41:58.607+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:41:58.608699+0000)
, 2022-04-03T00:41:58.665+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:41:58.873+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.591 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:09.611+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:09.612753+0000)
, 2022-04-03T00:42:09.880+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:42:09.922+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:09.923087+0000)

2022-04-03T14:13:12.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:20.654 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:42:20.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:42:20.819+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.593 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:31.622+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:42:31.931+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:31.932038+0000)
, 2022-04-03T00:42:31.993+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.593 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:42.255+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:42:42.295+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:42:42.527+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:42:53.091+0000 7fa9c620e700 -1 osd.4 63 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:42:53.160+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:42:53.160891+0000)
, 2022-04-03T00:42:53.422+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:04.230+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:43:04.379+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:43:04.626+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:15.623+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:43:15.635+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:15.635888+0000)
, 2022-04-03T00:43:15.823+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:26.575+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:43:26.639+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:26.639991+0000)
, 2022-04-03T00:43:26.756+0000 7f7b589b7700 -1 osd.5 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.597 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:37.643+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:37.644089+0000)
, 2022-04-03T00:43:37.815+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:43:37.830+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:48.688+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:43:48.732+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:43:48.961+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:48.961702+0000)

2022-04-03T14:13:12.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:43:59.729+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:43:59.741+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:43:59.965+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:43:59.966195+0000)

2022-04-03T14:13:12.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:10.823+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:44:10.940+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:44:10.969+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:10.970428+0000)

2022-04-03T14:13:12.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:21.660+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:21.660632+0000)
, 2022-04-03T00:44:21.774+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:44:21.783+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:32.745+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:44:32.801+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:44:32.978+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:32.979229+0000)

2022-04-03T14:13:12.603 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:43.983+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:43.983602+0000)
, 2022-04-03T00:44:44.176+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:44:44.203+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:44.203866+0000)

2022-04-03T14:13:12.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:44:54.670+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:54.672192+0000)
, 2022-04-03T00:44:54.968+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:44:54.986+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:44:54.987593+0000)

2022-04-03T14:13:12.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:05.701+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:45:05.990+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:05.991726+0000)
, 2022-04-03T00:45:06.211+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:06.212638+0000)

2022-04-03T14:13:12.605 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:16.995+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:16.995736+0000)
, 2022-04-03T00:45:17.216+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:17.216679+0000)
, 2022-04-03T00:45:17.381+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:27.999+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:28.000074+0000)
, 2022-04-03T00:45:28.187+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:45:28.219+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:28.221112+0000)

2022-04-03T14:13:12.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:39.051+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:45:39.187+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:45:39.225+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:39.225854+0000)

2022-04-03T14:13:12.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:45:50.158+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:45:50.229+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:50.230201+0000)
, 2022-04-03T00:45:50.691+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:45:50.692920+0000)

2022-04-03T14:13:12.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:01.011+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:01.012116+0000)
, 2022-04-03T00:46:01.179+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:46:01.233+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:01.234600+0000)

2022-04-03T14:13:12.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:12.237+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:12.239201+0000)
, 2022-04-03T00:46:12.580+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:46:12.652+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:23.185+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:46:23.242+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:23.243164+0000)
, 2022-04-03T00:46:23.575+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:34.247+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:34.247402+0000)
, 2022-04-03T00:46:34.677+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:46:34.708+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:34.709208+0000)

2022-04-03T14:13:12.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:45.664+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:46:45.712+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:45.713121+0000)
, 2022-04-03T00:46:45.785+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:46:56.541+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:46:56.709+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:46:56.716+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:46:56.717361+0000)

2022-04-03T14:13:12.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:07.720+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:07.721354+0000)
, 2022-04-03T00:47:07.772+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:47:07.789+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:18.742+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:47:18.787+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:47:18.849+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:29.728+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:29.729201+0000)
, 2022-04-03T00:47:29.760+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:47:29.944+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:40.801+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:47:40.879+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:47:40.884+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:47:51.736+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:47:51.737342+0000)
, 2022-04-03T00:47:51.769+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:47:51.888+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:02.787+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:48:02.836+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:48:02.949+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:14.028+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:48:14.061+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:14.061720+0000)
, 2022-04-03T00:48:14.112+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:24.940+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:48:25.007+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:48:25.065+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:25.065867+0000)

2022-04-03T14:13:12.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:36.031+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:48:36.038+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:48:36.069+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:36.070197+0000)

2022-04-03T14:13:12.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:47.020+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:48:47.073+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:47.074386+0000)
, 2022-04-03T00:48:47.112+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:48:58.076+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:48:58.078082+0000)
, 2022-04-03T00:48:58.100+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:48:58.111+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:09.123+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:09.136+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:09.309+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:09.310485+0000)

2022-04-03T14:13:12.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:20.171+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:20.313+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:20.314695+0000)
, 2022-04-03T00:49:20.748+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:31.270+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:31.318+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:31.319019+0000)
, 2022-04-03T00:49:31.326+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:42.440+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:42.634+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:42.777+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:49:42.778567+0000)

2022-04-03T14:13:12.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:49:53.346+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:53.492+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:49:53.613+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.624 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:04.384+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:50:04.654+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:50:04.692+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.625 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:15.732+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:50:15.768+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:50:15.789+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:15.790402+0000)

2022-04-03T14:13:12.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:26.562+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:50:26.612+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:50:26.794+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:26.794520+0000)

2022-04-03T14:13:12.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:37.637+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:50:37.797+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:37.798535+0000)
, 2022-04-03T00:50:37.803+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.627 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:48.713+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:50:48.801+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:48.802507+0000)
, 2022-04-03T00:50:49.123+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:49.124596+0000)

2022-04-03T14:13:12.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:50:59.806+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:50:59.806748+0000)
, 2022-04-03T00:50:59.881+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:51:00.127+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:00.128942+0000)

2022-04-03T14:13:12.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:10.846+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:51:11.132+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:11.133553+0000)
, 2022-04-03T00:51:11.358+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:11.358569+0000)

2022-04-03T14:13:12.629 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:22.137+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:22.137905+0000)
, 2022-04-03T00:51:22.252 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:51:22.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:12.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:32.947+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:51:33.141+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:33.142271+0000)
, 2022-04-03T00:51:33.367+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:33.368163+0000)

2022-04-03T14:13:12.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:44.304+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:51:44.371+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:44.372601+0000)
, 2022-04-03T00:51:44.532+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:51:55.150+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:55.151526+0000)
, 2022-04-03T00:51:55.262+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:51:55.376+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:51:55.377121+0000)

2022-04-03T14:13:12.632 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:06.256+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:52:06.380+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:06.381425+0000)
, 2022-04-03T00:52:06.398+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:17.314+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:52:17.385+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:17.385852+0000)
, 2022-04-03T00:52:17.431+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:28.390+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:52:28.390349+0000)
, 2022-04-03T00:52:28.426+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:52:28.596+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:39.395+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:52:39.568+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:52:39.647+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.635 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:52:50.652+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:52:50.666+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:52:50.724+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:01.403+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:01.403888+0000)
, 2022-04-03T00:53:01.706+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:01.706+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.637 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:12.583+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:12.775+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:12.853+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:12.854862+0000)

2022-04-03T14:13:12.638 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:23.489+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:53:23.678+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:23.808+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.639 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:34.698+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:34.861+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:34.862916+0000)
, 2022-04-03T00:53:34.879+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.640 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:45.859+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:45.866+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:45.867114+0000)
, 2022-04-03T00:53:46.110+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:53:56.841+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:53:56.870+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:56.871063+0000)
, 2022-04-03T00:53:57.198+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:53:57.198954+0000)

2022-04-03T14:13:12.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:07.927+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:54:08.202+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:08.203334+0000)
, 2022-04-03T00:54:08.278+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:19.162+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:54:19.207+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:19.207653+0000)
, 2022-04-03T00:54:19.264+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:30.210+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:30.212064+0000)
, 2022-04-03T00:54:30.246+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:54:30.293+0000 7f7b589b7700 -1 osd.5 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:40.322+0000 7fa9c620e700 -1 osd.4 64 get_health_metrics reporting 31 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:54:40.369+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:54:40.386+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.645 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:54:51.360+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:54:51.439+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:54:51.446+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:54:51.447097+0000)

2022-04-03T14:13:12.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:02.378+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:55:02.403+0000 7f7b589b7700 -1 osd.5 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:02.450+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:02.451077+0000)

2022-04-03T14:13:12.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:13.505+0000 7f7b589b7700 -1 osd.5 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:13.670+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:13.758+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:24.459+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:24.459672+0000)
, 2022-04-03T00:55:24.461+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:55:24.751+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:35.462+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:35.463859+0000)
, 2022-04-03T00:55:35.723+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:35.804+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:46.632+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:46.768+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:46.910+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:46.911093+0000)

2022-04-03T14:13:12.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:55:57.569+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:57.760+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:55:57.914+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:55:57.915571+0000)

2022-04-03T14:13:12.650 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:08.919+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:08.919340+0000)
, 2022-04-03T00:56:08.983+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:56:09.036+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:20.027+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:56:20.058+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:56:20.164+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:30.926+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:30.927361+0000)
, 2022-04-03T00:56:31.025+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:56:31.083+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:42.117+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:56:42.177+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:56:42.226+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:56:53.208+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:56:53.270+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:56:53.271425+0000)
, 2022-04-03T00:56:53.383+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:04.120+0000 7fa9c620e700 -1 osd.4 66 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:57:04.275+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:04.275791+0000)
, 2022-04-03T00:57:04.353+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.663 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:15.363+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:57:15.392+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:57:15.439+0000 7f7b589b7700 -1 osd.5 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:26.283+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:26.284667+0000)
, 2022-04-03T00:57:26.303+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:57:26.347+0000 7f7b589b7700 -1 osd.5 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.666 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:37.288+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:37.289485+0000)
, 2022-04-03T00:57:37.315+0000 7f7b589b7700 -1 osd.5 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:57:37.354+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.667 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:47.877+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:57:47.955+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:47.956228+0000)
, 2022-04-03T00:57:48.065+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:57:58.927+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:57:58.958+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:57:58.960023+0000)
, 2022-04-03T00:57:59.074+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:10.012+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:58:10.045+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:58:10.130+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.670 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:20.968+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:20.968336+0000)
, 2022-04-03T00:58:21.042+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:58:21.063+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.671 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:31.954+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:58:31.961+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:58:31.971+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:58:31.972716+0000)

2022-04-03T14:13:12.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:43.051+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:58:43.118+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T00:58:43.166+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:58:54.056 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:58:54.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T00:58:54.104+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:05.004+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:59:05.017+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:59:05.035+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:16.095+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:59:16.158+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:59:16.332+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:16.332255+0000)

2022-04-03T14:13:12.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:27.080+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T00:59:27.264+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:59:27.335+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:27.336752+0000)

2022-04-03T14:13:12.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:38.288+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T00:59:38.340+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:38.341466+0000)
, 2022-04-03T00:59:38.560+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:38.560867+0000)

2022-04-03T14:13:12.676 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T00:59:49.344+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:49.345821+0000)
, 2022-04-03T00:59:49.564+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-02T23:59:49.565008+0000)
, 2022-04-03T00:59:49.614+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:00.350+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:00.350640+0000)
, 2022-04-03T01:00:00.521+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:00:00.569+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:00.569547+0000)

2022-04-03T14:13:12.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:11.397+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:00:11.573+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:11.573919+0000)
, 2022-04-03T01:00:11.609+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:22.575+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:00:22.577+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:22.578607+0000)
, 2022-04-03T01:00:22.598+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.679 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:33.445+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:00:33.484+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:00:33.524+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:44.475+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:00:44.586+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:44.586773+0000)
, 2022-04-03T01:00:45.020+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:45.021100+0000)

2022-04-03T14:13:12.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:00:55.485+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:00:55.485+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:00:55.590+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:00:55.591259+0000)

2022-04-03T14:13:12.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:06.548+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:01:06.595+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:06.595756+0000)
, 2022-04-03T01:01:07.029+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:07.029690+0000)

2022-04-03T14:13:12.682 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:17.599+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:17.600174+0000)
, 2022-04-03T01:01:18.032+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:18.033826+0000)
, 2022-04-03T01:01:18.175+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:28.603+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:28.604470+0000)
, 2022-04-03T01:01:29.036+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:29.037758+0000)
, 2022-04-03T01:01:29.069+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:40.041+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:01:40.041768+0000)
, 2022-04-03T01:01:40.164+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:01:40.224+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:01:51.198+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:01:51.206+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:01:51.299+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.685 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:02.049+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:02:02.050147+0000)
, 2022-04-03T01:02:02.084+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:02:02.227+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:13.249+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:02:13.404+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:02:13.405114+0000)
, 2022-04-03T01:02:13.405+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:24.315+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:02:24.337+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:24.339+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:35.292+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:35.293+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:02:35.299+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:46.325+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:46.328+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:02:46.384+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:02:57.165+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:02:57.225+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:02:57.381+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:08.415+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:03:08.427+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:08.427165+0000)
, 2022-04-03T01:03:08.526+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:19.430+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:19.431697+0000)
, 2022-04-03T01:03:19.515+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:03:19.650+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:19.651825+0000)

2022-04-03T14:13:12.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:30.435+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:30.435988+0000)
, 2022-04-03T01:03:30.516+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:03:30.654+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:30.655958+0000)

2022-04-03T14:13:12.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:41.473+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:03:41.618+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:03:41.659+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:41.660440+0000)

2022-04-03T14:13:12.692 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:03:52.664+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:03:52.664655+0000)
, 2022-04-03T01:03:52.762+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:03:52.889+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:03.491+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:04:03.668+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:03.669150+0000)
, 2022-04-03T01:04:03.844+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:14.771+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:04:14.844+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:04:15.100+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:15.101284+0000)

2022-04-03T14:13:12.694 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:25.677+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:25.678156+0000)
, 2022-04-03T01:04:25.809+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:04:25.942+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.695 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:36.717+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:04:36.995+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:04:37.109+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:37.109743+0000)

2022-04-03T14:13:12.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:47.924+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:04:48.113+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:48.114125+0000)
, 2022-04-03T01:04:48.126+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:04:58.855+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:04:59.118+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:04:59.118388+0000)
, 2022-04-03T01:04:59.141+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.697 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:10.121+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:10.122375+0000)
, 2022-04-03T01:05:10.274+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:05:10.298+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.698 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:21.316+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:05:21.399+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:05:21.433+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.698 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:32.130+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:32.131426+0000)
, 2022-04-03T01:05:32.262+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:05:32.398+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.699 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:43.395+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:05:43.489+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:43.489497+0000)
, 2022-04-03T01:05:43.490+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.700 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:05:54.460+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:05:54.493+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:05:54.494004+0000)
, 2022-04-03T01:05:54.572+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:05.280+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:06:05.497+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:05.498156+0000)
, 2022-04-03T01:06:05.640+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:16.501+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:16.502624+0000)
, 2022-04-03T01:06:16.661+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:06:16.721+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:16.722716+0000)

2022-04-03T14:13:12.702 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:27.505+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:27.507255+0000)
, 2022-04-03T01:06:27.726+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:27.726895+0000)
, 2022-04-03T01:06:27.796+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.703 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:38.730+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:06:38.731346+0000)
, 2022-04-03T01:06:38.737+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:06:38.796+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.704 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:06:49.844+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:06:49.875+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:06:50.014+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.704 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:00.813+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:07:00.870+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:07:00.902+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.705 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:11.903+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:07:12.156+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:07:12.170+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:12.171483+0000)

2022-04-03T14:13:12.706 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:23.175+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:23.175885+0000)
, 2022-04-03T01:07:23.241+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:07:23.373+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:33.863+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:07:34.179+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:34.180241+0000)
, 2022-04-03T01:07:34.358+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:45.183+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:45.184094+0000)
, 2022-04-03T01:07:45.407+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:07:45.451+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.708 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:07:56.188+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:07:56.188479+0000)
, 2022-04-03T01:07:56.283+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:07:56.355+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:07.375+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:08:07.490+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:08:07.510+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:18.432+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:08:18.545+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:08:18.551+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:18.551965+0000)

2022-04-03T14:13:12.710 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:29.433+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:08:29.555+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:29.556567+0000)
, 2022-04-03T01:08:29.565+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:40.560+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:40.561180+0000)
, 2022-04-03T01:08:40.589+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:08:40.777+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:40.779087+0000)

2022-04-03T14:13:12.712 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:08:51.565+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:51.565930+0000)
, 2022-04-03T01:08:51.783+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:08:51.783680+0000)
, 2022-04-03T01:08:51.896+0000 7f7b589b7700 -1 osd.5 67 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:02.569+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:02.570148+0000)
, 2022-04-03T01:09:02.613+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:09:02.787+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:02.788112+0000)

2022-04-03T14:13:12.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:13.684+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:09:13.782+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:13.783399+0000)
, 2022-04-03T01:09:13.792+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:13.792856+0000)

2022-04-03T14:13:12.714 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:23.584+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:09:23.785+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:23.786620+0000)
, 2022-04-03T01:09:23.796+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:23.796768+0000)

2022-04-03T14:13:12.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:33.380+0000 7fa9c620e700 -1 osd.4 67 get_health_metrics reporting 32 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:09:33.582+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:33.583079+0000)
, 2022-04-03T01:09:33.638+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.716 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:42.586+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:42.586670+0000)
, 2022-04-03T01:09:42.756+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:09:42.793+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:42.793486+0000)

2022-04-03T14:13:12.716 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:09:52.590+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:52.590710+0000)
, 2022-04-03T01:09:52.796+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:52.796611+0000)
, 2022-04-03T01:09:52.808+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:09:52.808755+0000)

2022-04-03T14:13:12.717 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:02.245+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:10:02.298+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:10:02.594+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:02.595017+0000)

2022-04-03T14:13:12.718 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:12.390+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:10:12.598+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:12.599303+0000)
, 2022-04-03T01:10:12.802+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:12.803367+0000)

2022-04-03T14:13:12.719 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:22.362+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:10:22.602+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:22.603395+0000)
, 2022-04-03T01:10:22.806+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:22.806670+0000)

2022-04-03T14:13:12.719 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:32.189+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:10:32.195+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:10:32.250+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:32.250646+0000)

2022-04-03T14:13:12.720 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:42.165+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:10:42.254+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:42.254513+0000)
, 2022-04-03T01:10:42.611+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:42.611357+0000)

2022-04-03T14:13:12.721 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:10:52.257+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:52.258272+0000)
, 2022-04-03T01:10:52.328+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:10:52.615+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:10:52.615601+0000)

2022-04-03T14:13:12.721 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:02.187+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:11:02.205+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:11:02.260+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:02.261968+0000)

2022-04-03T14:13:12.722 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:12.122+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:11:12.258+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:11:12.265+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:12.265976+0000)

2022-04-03T14:13:12.723 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:22.087+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:11:22.269+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:22.270209+0000)
, 2022-04-03T01:11:22.337+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.724 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:31.948+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:11:32.048+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:11:32.273+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:32.274033+0000)

2022-04-03T14:13:12.724 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:42.004+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:11:42.277+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:11:42.277918+0000)
, 2022-04-03T01:11:42.339+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.725 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:11:52.019+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:11:52.212+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:11:52.259+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.726 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:01.860+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:01.861159+0000)
, 2022-04-03T01:12:01.914+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:02.092+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.726 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:12.038+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:12.148+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:12.268+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.727 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:22.047+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:22.229+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:12:22.278+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.728 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:31.873+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:31.873601+0000)
, 2022-04-03T01:12:31.875+0000 7f7b589b7700 -1 osd.5 69 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:31.960+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.729 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:41.881+0000 7f7b589b7700 -1 osd.5 69 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:42.051+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:12:42.300+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:42.300378+0000)

2022-04-03T14:13:12.729 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:12:51.663+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:12:51.664577+0000)
, 2022-04-03T01:12:51.815+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:12:51.830+0000 7f7b589b7700 -1 osd.5 69 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.730 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:01.393+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:13:01.401+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:01.669+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:01.670719+0000)

2022-04-03T14:13:12.731 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:11.462+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:11.623+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:13:11.673+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:11.674507+0000)

2022-04-03T14:13:12.731 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:21.484+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:21.675+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:13:21.678+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:21.678732+0000)

2022-04-03T14:13:12.732 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:31.441+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:13:31.500+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:31.526+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.733 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:41.501+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:41.605+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:13:41.660+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.734 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:13:51.498+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:13:51.592+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:13:51.689+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:13:51.691072+0000)

2022-04-03T14:13:12.734 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:01.328+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:01.329801+0000)
, 2022-04-03T01:14:01.471+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:14:01.510+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.735 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:11.332+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:11.333598+0000)
, 2022-04-03T01:14:11.419+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:11.513+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.736 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:21.404+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:21.406+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:14:21.484+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.736 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:31.224+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:14:31.325+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:31.341+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:31.341371+0000)

2022-04-03T14:13:12.737 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:41.273+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:14:41.325+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:14:41.344+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:41.345105+0000)

2022-04-03T14:13:12.738 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:14:51.291+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:14:51.347+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:14:51.348890+0000)
, 2022-04-03T01:14:51.419+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:00.939+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:15:01.155+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:01.251+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.740 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:11.178+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:11.211+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:11.340+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.740 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:21.194+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:21.227+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:21.323+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:30.941+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:15:30.946+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:15:30.946708+0000)
, 2022-04-03T01:15:31.209+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.742 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:41.024+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:15:41.195+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:41.257+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.742 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:15:51.056+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:15:51.058+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:15:51.299+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:00.921+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:16:00.942+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:16:00.958+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:00.959376+0000)

2022-04-03T14:13:12.744 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:10.933+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:16:10.962+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:10.963503+0000)
, 2022-04-03T01:16:11.023+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.745 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:20.967+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:20.967557+0000)
, 2022-04-03T01:16:20.988+0000 7fa9c620e700 -1 osd.4 69 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:16:21.078+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.745 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:30.931+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:30.931658+0000)
, 2022-04-03T01:16:30.944+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:16:30.971+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:30.971785+0000)

2022-04-03T14:13:12.746 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:40.934+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:40.934893+0000)
, 2022-04-03T01:16:40.975+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:40.975976+0000)
, 2022-04-03T01:16:41.096+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.747 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:16:50.937+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:50.938327+0000)
, 2022-04-03T01:16:50.979+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:16:50.980106+0000)
, 2022-04-03T01:16:51.206+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:00.768+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:00.770161+0000)
, 2022-04-03T01:17:00.940+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:00.941444+0000)
, 2022-04-03T01:17:00.945+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:10.773+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:10.774253+0000)
, 2022-04-03T01:17:10.944+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:10.944966+0000)
, 2022-04-03T01:17:10.988+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:10.988776+0000)

2022-04-03T14:13:12.749 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:20.777+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:20.778394+0000)
, 2022-04-03T01:17:20.948+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:20.948588+0000)
, 2022-04-03T01:17:20.992+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:20.992882+0000)

2022-04-03T14:13:12.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:30.727+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:17:30.782+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:30.782670+0000)
, 2022-04-03T01:17:30.947+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:40.772+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:17:40.785+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:40.786720+0000)
, 2022-04-03T01:17:40.954+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:17:40.955320+0000)

2022-04-03T14:13:12.751 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:17:50.612+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:17:50.673+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:17:50.739+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.752 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:00.451+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:00.709+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:00.715+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:10.508+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:10.749+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:18:10.753+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:20.624+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:20.646+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:20.798+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.754 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:30.436+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:30.436448+0000)
, 2022-04-03T01:18:30.560+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:30.664+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:40.439+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:40.440867+0000)
, 2022-04-03T01:18:40.476+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:40.617+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:18:50.444+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:18:50.445066+0000)
, 2022-04-03T01:18:50.553+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:18:50.704+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.756 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:00.265+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:19:00.448+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:00.448996+0000)
, 2022-04-03T01:19:00.499+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.757 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:10.423+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:19:10.452+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:10.453183+0000)
, 2022-04-03T01:19:10.506+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:20.456+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:20.457185+0000)
, 2022-04-03T01:19:20.576+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:19:20.670+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:30.242+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:19:30.461+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:30.461222+0000)
, 2022-04-03T01:19:30.581+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.759 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:40.464+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:40.465335+0000)
, 2022-04-03T01:19:40.474+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:19:40.668+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.760 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:19:50.468+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:19:50.469686+0000)
, 2022-04-03T01:19:50.478+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:19:50.611+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.760 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:00.058+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:00.059265+0000)
, 2022-04-03T01:20:00.443+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:20:00.472+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:00.473647+0000)

2022-04-03T14:13:12.761 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:10.063+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:10.063620+0000)
, 2022-04-03T01:20:10.477+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:10.477889+0000)
, 2022-04-03T01:20:10.548+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.762 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:20.227+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:20:20.479+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:20:20.481+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:20.482032+0000)

2022-04-03T14:13:12.763 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:30.071+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:30.071808+0000)
, 2022-04-03T01:20:30.141+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:20:30.471+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.763 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:40.074+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:40.075853+0000)
, 2022-04-03T01:20:40.220+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:20:40.372+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.764 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:20:50.079+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:20:50.079927+0000)
, 2022-04-03T01:20:50.144+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:20:50.338+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.765 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:00.021+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:00.022533+0000)
, 2022-04-03T01:21:00.083+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:00.083945+0000)
, 2022-04-03T01:21:00.084+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.765 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:10.025+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:10.025847+0000)
, 2022-04-03T01:21:10.087+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:10.088311+0000)
, 2022-04-03T01:21:10.126+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.766 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:20.028+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:20.029153+0000)
, 2022-04-03T01:21:20.092+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:20.092487+0000)
, 2022-04-03T01:21:20.152+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.767 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:30.013+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:21:30.032+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:30.032679+0000)
, 2022-04-03T01:21:30.096+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:30.096903+0000)

2022-04-03T14:13:12.767 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:39.886+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:39.887871+0000)
, 2022-04-03T01:21:40.034+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:40.036046+0000)
, 2022-04-03T01:21:40.100+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:40.101272+0000)

2022-04-03T14:13:12.768 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:49.891+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:49.892022+0000)
, 2022-04-03T01:21:50.038+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:50.039391+0000)
, 2022-04-03T01:21:50.104+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:50.105137+0000)

2022-04-03T14:13:12.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:21:59.746+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:21:59.895+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:21:59.896409+0000)
, 2022-04-03T01:22:00.042+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:00.042853+0000)

2022-04-03T14:13:12.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:09.801+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:22:09.900+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:09.900921+0000)
, 2022-04-03T01:22:10.045+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:10.046053+0000)

2022-04-03T14:13:12.770 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:19.866+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:22:19.904+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:19.905200+0000)
, 2022-04-03T01:22:20.049+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:20.049450+0000)

2022-04-03T14:13:12.771 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:29.791+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:22:29.908+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:29.909341+0000)
, 2022-04-03T01:22:29.938+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.772 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:39.711+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:22:39.912+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:39.913717+0000)
, 2022-04-03T01:22:40.010+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.772 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:49.742+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:22:49.917+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:49.918166+0000)
, 2022-04-03T01:22:50.047+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.773 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:22:59.545+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:59.545784+0000)
, 2022-04-03T01:22:59.692+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:22:59.922+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:22:59.923085+0000)

2022-04-03T14:13:12.774 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:09.549+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:09.549857+0000)
, 2022-04-03T01:23:09.700+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:23:09.926+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:09.927286+0000)

2022-04-03T14:13:12.774 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:19.536+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:23:19.553+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:19.553785+0000)
, 2022-04-03T01:23:19.931+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:19.932005+0000)

2022-04-03T14:13:12.775 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:29.421+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:23:29.556+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:29.557878+0000)
, 2022-04-03T01:23:29.585+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.776 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:39.529+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:23:39.560+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:39.561883+0000)
, 2022-04-03T01:23:39.722+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.777 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:49.565+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:49.565991+0000)
, 2022-04-03T01:23:49.597+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:23:49.628+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.777 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:23:59.468+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:23:59.569+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:23:59.570208+0000)
, 2022-04-03T01:23:59.644+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.778 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:09.417+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:09.573+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:09.574261+0000)
, 2022-04-03T01:24:09.599+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.779 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:19.418+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:24:19.568+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:24:19.578+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:19.578438+0000)

2022-04-03T14:13:12.779 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:29.495+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:29.501+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:24:29.581+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:29.582674+0000)

2022-04-03T14:13:12.780 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:39.322+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:39.574+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:39.585+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:39.587139+0000)

2022-04-03T14:13:12.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:48.329+0000 7fa9c620e700 -1 osd.4 70 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:48.432+0000 7f7b589b7700 -1 osd.5 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:48.589+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:48.590556+0000)

2022-04-03T14:13:12.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:24:58.343+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:24:58.594+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:24:58.594763+0000)
, 2022-04-03T01:24:58.738+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.782 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:08.179+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:25:08.191+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:08.191698+0000)
, 2022-04-03T01:25:08.300+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.783 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:18.199+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:25:18.326+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:25:18.378+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:28.199+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:28.200835+0000)
, 2022-04-03T01:25:28.432+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:25:28.606+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:28.607508+0000)

2022-04-03T14:13:12.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:38.153+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:25:38.162+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:25:38.204+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:38.205127+0000)

2022-04-03T14:13:12.785 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:48.121+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:48.122647+0000)
, 2022-04-03T01:25:48.208+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:48.209457+0000)
, 2022-04-03T01:25:48.222+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:25:58.213+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:25:58.213833+0000)
, 2022-04-03T01:25:58.253+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:25:58.580+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:08.129+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:08.130161+0000)
, 2022-04-03T01:26:08.168+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:26:08.183+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.787 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:18.089+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:26:18.133+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:18.133780+0000)
, 2022-04-03T01:26:18.154+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:28.136+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:28.137448+0000)
, 2022-04-03T01:26:28.224+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:26:28.227+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:28.227980+0000)

2022-04-03T14:13:12.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:38.017+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:38.018513+0000)
, 2022-04-03T01:26:38.031+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:26:38.139+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:38.141169+0000)

2022-04-03T14:13:12.789 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:48.002+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:26:48.022+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:48.022881+0000)
, 2022-04-03T01:26:48.143+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:48.144785+0000)

2022-04-03T14:13:12.790 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:26:58.102+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:26:58.147+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:58.148281+0000)
, 2022-04-03T01:26:58.240+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:26:58.241087+0000)

2022-04-03T14:13:12.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:07.996+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:08.013+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:08.030+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:08.030583+0000)

2022-04-03T14:13:12.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:17.924+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:18.014+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:18.034+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:18.035030+0000)

2022-04-03T14:13:12.792 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:28.030+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:28.038+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:28.039484+0000)
, 2022-04-03T01:27:28.158+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:28.159254+0000)

2022-04-03T14:13:12.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:37.777+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:37.836+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:38.042+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:38.043745+0000)

2022-04-03T14:13:12.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:47.697+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:47.923+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:47.963+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.794 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:27:57.963+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:57.989+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:27:58.051+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:27:58.052331+0000)

2022-04-03T14:13:12.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:07.580+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:28:07.674+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:07.674707+0000)
, 2022-04-03T01:28:07.957+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:17.677+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:17.678972+0000)
, 2022-04-03T01:28:17.725+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:28:17.893+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.796 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:27.740+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:28:27.917+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:28:28.063+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:28.064513+0000)

2022-04-03T14:13:12.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:37.634+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:37.685+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:37.686513+0000)
, 2022-04-03T01:28:37.700+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:47.669+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:28:47.689+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:47.690519+0000)
, 2022-04-03T01:28:47.798+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:28:57.694+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:28:57.694745+0000)
, 2022-04-03T01:28:57.718+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:28:58.013+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:07.571+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:29:07.607+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:07.698+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:07.698627+0000)

2022-04-03T14:13:12.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:17.618+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:17.678+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:29:17.701+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:17.703032+0000)

2022-04-03T14:13:12.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:27.671+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:27.707+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:27.708662+0000)
, 2022-04-03T01:29:27.910+0000 7f7b589b7700 -1 osd.5 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:37.634+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:37.646+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:37.712+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:37.712795+0000)

2022-04-03T14:13:12.802 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:47.638+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:47.654+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:47.716+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:47.717070+0000)

2022-04-03T14:13:12.803 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:29:57.659+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:29:57.697+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:29:57.720+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:29:57.721053+0000)

2022-04-03T14:13:12.803 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:07.495+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:30:07.565+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:30:07.641+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:17.391+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:30:17.724+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:30:17.728+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:17.729148+0000)

2022-04-03T14:13:12.805 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:27.630+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:30:27.662+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:30:27.732+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:27.733210+0000)

2022-04-03T14:13:12.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:37.271+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:30:37.333+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:37.334469+0000)
, 2022-04-03T01:30:37.513+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:47.271+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:30:47.338+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:47.339089+0000)
, 2022-04-03T01:30:47.513+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.807 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:30:57.343+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:30:57.343476+0000)
, 2022-04-03T01:30:57.488+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:30:57.607+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:07.236+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:07.236472+0000)
, 2022-04-03T01:31:07.262+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:31:07.347+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:07.348062+0000)

2022-04-03T14:13:12.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:17.239+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:17.240150+0000)
, 2022-04-03T01:31:17.331+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:31:17.351+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:17.352499+0000)

2022-04-03T14:13:12.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:27.355+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:27.356706+0000)
, 2022-04-03T01:31:27.396+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:31:27.450+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.810 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:37.147+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:37.148416+0000)
, 2022-04-03T01:31:37.246+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:37.247395+0000)
, 2022-04-03T01:31:37.360+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:37.361430+0000)

2022-04-03T14:13:12.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:47.152+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:47.152650+0000)
, 2022-04-03T01:31:47.250+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:47.250693+0000)
, 2022-04-03T01:31:47.365+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:47.366055+0000)

2022-04-03T14:13:12.812 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:31:57.254+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:57.254459+0000)
, 2022-04-03T01:31:57.370+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:31:57.370857+0000)
, 2022-04-03T01:31:57.482+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.812 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:07.033+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:32:07.161+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:07.161260+0000)
, 2022-04-03T01:32:07.257+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:07.257837+0000)

2022-04-03T14:13:12.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:17.116+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:32:17.164+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:17.165674+0000)
, 2022-04-03T01:32:17.260+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:17.261372+0000)

2022-04-03T14:13:12.814 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:27.201+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:32:27.263+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:27.265117+0000)
, 2022-04-03T01:32:27.383+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:27.384523+0000)

2022-04-03T14:13:12.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:37.010+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:32:37.160+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:32:37.173+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:37.174362+0000)

2022-04-03T14:13:12.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:47.001+0000 7fa9c620e700 -1 osd.4 72 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:32:47.177+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:47.178222+0000)
, 2022-04-03T01:32:47.272+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:47.272544+0000)

2022-04-03T14:13:12.816 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:32:56.679+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:32:56.794+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:32:56.794676+0000)
, 2022-04-03T01:32:56.916+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:06.602+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:33:06.622+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:06.670+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.818 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:16.511+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:16.600+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:33:16.679+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.818 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:26.725+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:33:26.763+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:33:26.805+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:26.806950+0000)

2022-04-03T14:13:12.819 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:36.414+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:36.414784+0000)
, 2022-04-03T01:33:36.443+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:36.696+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.820 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:46.419+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:46.419421+0000)
, 2022-04-03T01:33:46.479+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:46.631+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.821 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:33:56.549+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:33:56.579+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:33:56.818+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:33:56.818735+0000)

2022-04-03T14:13:12.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:06.305+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:34:06.381+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:34:06.427+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:06.428164+0000)

2022-04-03T14:13:12.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:16.418+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:34:16.431+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:16.432399+0000)
, 2022-04-03T01:34:16.439+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:26.436+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:26.436691+0000)
, 2022-04-03T01:34:26.491+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:34:26.760+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:36.310+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:36.312146+0000)
, 2022-04-03T01:34:36.351+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:34:36.440+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:36.441014+0000)

2022-04-03T14:13:12.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:46.315+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:46.315878+0000)
, 2022-04-03T01:34:46.434+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:34:46.445+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:46.445814+0000)

2022-04-03T14:13:12.826 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:34:56.449+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:34:56.449769+0000)
, 2022-04-03T01:34:56.539+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:34:56.606+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.827 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:06.238+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:06.239141+0000)
, 2022-04-03T01:35:06.321+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:06.322799+0000)
, 2022-04-03T01:35:06.453+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:06.454431+0000)

2022-04-03T14:13:12.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:16.262+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:35:16.325+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:16.326564+0000)
, 2022-04-03T01:35:16.458+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:16.459117+0000)

2022-04-03T14:13:12.829 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:26.329+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:26.330140+0000)
, 2022-04-03T01:35:26.463+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:26.463867+0000)
, 2022-04-03T01:35:26.580+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.829 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:36.251+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:36.252382+0000)
, 2022-04-03T01:35:36.295+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:35:36.333+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:36.333946+0000)

2022-04-03T14:13:12.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:46.257+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:46.257335+0000)
, 2022-04-03T01:35:46.337+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:46.337533+0000)
, 2022-04-03T01:35:46.343+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.831 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:35:56.293+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:35:56.313+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:35:56.340+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:35:56.340882+0000)

2022-04-03T14:13:12.832 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:06.102+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:36:06.265+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:06.266535+0000)
, 2022-04-03T01:36:06.343+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:06.344167+0000)

2022-04-03T14:13:12.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:16.033+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T01:36:16.270+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:16.270856+0000)
, 2022-04-03T01:36:16.346+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:16.347604+0000)

2022-04-03T14:13:12.834 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:26.211+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:36:26.274+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:26.275406+0000)
, 2022-04-03T01:36:26.350+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:26.350914+0000)

2022-04-03T14:13:12.834 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:35.887+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:35.887570+0000)
, 2022-04-03T01:36:35.961+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:36:36.058+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:45.911+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:36:45.997+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:36:46.036+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.836 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:36:55.944+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:36:56.087+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:36:56.287+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:36:56.288255+0000)

2022-04-03T14:13:12.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:05.741+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:05.850+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:05.898+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:05.899657+0000)

2022-04-03T14:13:12.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:15.891+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:15.903+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:15.903865+0000)
, 2022-04-03T01:37:16.037+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:12.838 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:25.853+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:25.907+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:25.907999+0000)
, 2022-04-03T01:37:26.141+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:35.430+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:35.521+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:35.522095+0000)
, 2022-04-03T01:37:35.727+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:45.526+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:45.526366+0000)
, 2022-04-03T01:37:45.698+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:45.742+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:37:55.529+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:37:55.530947+0000)
, 2022-04-03T01:37:55.572+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:37:55.712+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.841 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:05.405+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:05.453+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:38:05.534+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:05.535525+0000)

2022-04-03T14:13:12.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:15.539+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:15.539796+0000)
, 2022-04-03T01:38:15.550+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:15.605+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:25.497+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:25.543+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:25.544398+0000)
, 2022-04-03T01:38:25.609+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.843 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:29.060+0000 7fcc4e787700  1 --  start start
, 2022-04-03T01:38:29.061+0000 7fcc4e787700  1 --2-  >> v2:172.21.15.61:3300/0 conn(0x7fcc4814e8c0 0x7fcc480c6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect
, 2022-04-03T01:38:29.061+0000 7fcc4e787700  1 --2-  >> v2:172.21.15.67:3300/0 conn(0x7fcc4814ec90 0x7fcc48154b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect

2022-04-03T14:13:12.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:34.333+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:38:34.397+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:34.397372+0000)
, 2022-04-03T01:38:34.432+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:12.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:44.415+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:38:44.486+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:44.552+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:38:44.552592+0000)

2022-04-03T14:13:12.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:38:54.428+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:38:54.493+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:38:54.545+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:04.344+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:04.345047+0000)
, 2022-04-03T01:39:04.388+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:39:04.406+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:04.408148+0000)

2022-04-03T14:13:12.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:14.335+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:39:14.348+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:14.349660+0000)
, 2022-04-03T01:39:14.400+0000 7fa9c620e700 -1 osd.4 73 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:24.281+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:39:24.353+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:24.354224+0000)
, 2022-04-03T01:39:24.414+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:24.415051+0000)

2022-04-03T14:13:12.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:34.146+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:39:34.184+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:39:34.262+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:44.258+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:39:44.264+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:39:44.363+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:39:44.363584+0000)

2022-04-03T14:13:12.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:39:53.160 INFO:tasks.ceph.mds.b.smithi067.stdout:starting mds.b at
, 2022-04-03T01:39:53.191+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:39:53.233+0000 7f7b589b7700 -1 osd.5 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:02.800+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:40:02.956+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:40:02.974+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:02.975216+0000)

2022-04-03T14:13:12.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:12.948+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:40:12.978+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:12.978963+0000)
, 2022-04-03T01:40:13.181+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.854 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:22.982+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:22.982823+0000)
, 2022-04-03T01:40:23.030+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:40:23.333+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:12.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:29.382+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:29.383611+0000)
, 2022-04-03T01:40:29.437+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:29.438710+0000)
, 2022-04-03T01:40:29.532+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:12.856 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:29.996 INFO:teuthology.orchestra.run.smithi061.stdout:systemd       1                           root   97u     unix 0xffff888103940f00         0t0         17904 /run/systemd/journal/stdout type=STREAM
, 2022-04-03T01:40:29.996 INFO:teuthology.orchestra.run.smithi061.stdout:systemd       1                           root   98u     unix 0xffff88810e66a580         0t0           625 /run/systemd/journal/stdout type=STREAM
, 2022-04-03T01:40:29.996 INFO:teuthology.orchestra.run.smithi061.stdout:systemd       1                           root   99u     unix 0xffff888103865a00         0t0         18106 /run/systemd/journal/stdout type=STREAM

2022-04-03T14:13:12.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.030 INFO:teuthology.orchestra.run.smithi061.stdout:migration    38                           root  cwd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:30.030 INFO:teuthology.orchestra.run.smithi061.stdout:migration    38                           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:30.030 INFO:teuthology.orchestra.run.smithi061.stdout:migration    38                           root  txt   unknown                                              /proc/38/exe

2022-04-03T14:13:12.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.063 INFO:teuthology.orchestra.run.smithi061.stdout:kthrotld     81                           root  txt   unknown                                              /proc/81/exe
, 2022-04-03T01:40:30.064 INFO:teuthology.orchestra.run.smithi061.stdout:acpi_ther    84                           root  cwd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:30.064 INFO:teuthology.orchestra.run.smithi061.stdout:acpi_ther    84                           root  rtd       DIR                8,1        4096             2 /

2022-04-03T14:13:12.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.099 INFO:teuthology.orchestra.run.smithi061.stdout:kworker/5   266                           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:30.100 INFO:teuthology.orchestra.run.smithi061.stdout:kworker/5   266                           root  txt   unknown                                              /proc/266/exe
, 2022-04-03T01:40:30.100 INFO:teuthology.orchestra.run.smithi061.stdout:nvme-wq     401                           root  cwd       DIR                8,1        4096             2 /

2022-04-03T14:13:12.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.135 INFO:teuthology.orchestra.run.smithi061.stdout:rpcbind     606                            rpc  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:30.135 INFO:teuthology.orchestra.run.smithi061.stdout:rpcbind     606                            rpc  mem       REG                8,1       71480         10649 /usr/lib64/libkrb5support.so.0.1
, 2022-04-03T01:40:30.136 INFO:teuthology.orchestra.run.smithi061.stdout:rpcbind     606                            rpc  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1

2022-04-03T14:13:12.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.170 INFO:teuthology.orchestra.run.smithi061.stdout:auditd      610   611 auditd              root    9u     unix 0xffff88817f45da00         0t0           454 type=STREAM
, 2022-04-03T01:40:30.171 INFO:teuthology.orchestra.run.smithi061.stdout:auditd      610   611 auditd              root   10u     unix 0xffff88817f45cb00         0t0           455 type=STREAM
, 2022-04-03T01:40:30.171 INFO:teuthology.orchestra.run.smithi061.stdout:auditd      610   611 auditd              root   11u  a_inode               0,13           0         11317 [eventfd]

2022-04-03T14:13:12.861 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.205 INFO:teuthology.orchestra.run.smithi061.stdout:sssd        697                           root  mem       REG                8,1       71048         43211 /usr/lib64/ldb/modules/ldb/memberof.so
, 2022-04-03T01:40:30.206 INFO:teuthology.orchestra.run.smithi061.stdout:sssd        697                           root  mem       REG                8,1        7544         43353 /usr/lib64/ldb/modules/ldb/mdb.so
, 2022-04-03T01:40:30.206 INFO:teuthology.orchestra.run.smithi061.stdout:sssd        697                           root  mem       REG                8,1       91808         10184 /usr/lib64/liblmdb.so.0.0.0

2022-04-03T14:13:12.861 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.240 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698                        polkitd  mem       REG                8,1       83808          8748 /usr/lib64/libnss_files-2.28.so
, 2022-04-03T01:40:30.240 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698                        polkitd  DEL       REG                8,1                     65927 /var/lib/sss/mc/passwd
, 2022-04-03T01:40:30.240 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698                        polkitd  mem       REG                8,1       46312         13315 /usr/lib64/libnss_sss.so.2

2022-04-03T14:13:12.862 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.274 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   790 gmain            polkitd  mem       REG                8,1       26998         19541 /usr/lib64/gconv/gconv-modules.cache
, 2022-04-03T01:40:30.275 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   790 gmain            polkitd    0u      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:30.275 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   790 gmain            polkitd    1u      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:12.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.309 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   793 JS\x20Hel        polkitd  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1
, 2022-04-03T01:40:30.309 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   793 JS\x20Hel        polkitd  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:30.310 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   793 JS\x20Hel        polkitd  mem       REG                8,1      464936          9019 /usr/lib64/libpcre.so.1.2.10

2022-04-03T14:13:12.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.344 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   795 JS\x20Hel        polkitd  mem       REG                8,1      145984          8939 /usr/lib64/libgpg-error.so.0.24.2
, 2022-04-03T01:40:30.345 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   795 JS\x20Hel        polkitd  DEL       REG                8,1                      8915 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:30.345 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   795 JS\x20Hel        polkitd  DEL       REG                8,1                     10923 /usr/lib64/libblkid.so.1.1.0

2022-04-03T14:13:12.864 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.378 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   796 JS\x20Hel        polkitd    2u      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:30.379 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   796 JS\x20Hel        polkitd    3r      REG                8,1     9253600         65927 /var/lib/sss/mc/passwd (deleted)
, 2022-04-03T01:40:30.379 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   796 JS\x20Hel        polkitd    4r      REG                8,1    11567160         65929 /var/lib/sss/mc/initgroups (deleted)

2022-04-03T14:13:12.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.439 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   798 JS\x20Hel        polkitd  mem       REG                8,1     2051656         11543 /usr/lib64/libgnutls.so.30.28.2
, 2022-04-03T01:40:30.439 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   798 JS\x20Hel        polkitd  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:30.440 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   798 JS\x20Hel        polkitd  mem       REG                8,1       37232          9251 /usr/lib64/libffi.so.6.0.2

2022-04-03T14:13:12.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.475 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   800 JS\x20Hel        polkitd  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:30.475 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   800 JS\x20Hel        polkitd  mem       REG                8,1     1541856          9167 /usr/lib64/libgmp.so.10.3.2
, 2022-04-03T01:40:30.475 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   800 JS\x20Hel        polkitd  mem       REG                8,1      197728          9361 /usr/lib64/libhogweed.so.4.5

2022-04-03T14:13:12.866 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.509 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   818 polkitd          polkitd    5r  a_inode               0,13           0         11317 inotify
, 2022-04-03T01:40:30.509 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   818 polkitd          polkitd    6u  a_inode               0,13           0         11317 [eventfd]
, 2022-04-03T01:40:30.509 INFO:teuthology.orchestra.run.smithi061.stdout:polkitd     698   818 polkitd          polkitd    7u  a_inode               0,13           0         11317 [eventfd]

2022-04-03T14:13:12.867 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.543 INFO:teuthology.orchestra.run.smithi061.stdout:dbus-daem   699   771 dbus-daem           dbus   14u     unix 0xffff88810ebdf800         0t0         17232 type=DGRAM
, 2022-04-03T01:40:30.543 INFO:teuthology.orchestra.run.smithi061.stdout:dbus-daem   699   771 dbus-daem           dbus   15u     unix 0xffff88817f45f800         0t0         21294 /run/dbus/system_bus_socket type=STREAM
, 2022-04-03T01:40:30.544 INFO:teuthology.orchestra.run.smithi061.stdout:dbus-daem   699   771 dbus-daem           dbus   16r      REG                8,1     9253600         46453 /var/lib/sss/mc/passwd

2022-04-03T14:13:12.868 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.578 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700                           root    9r      REG                0,4           0    4026532498 mnt
, 2022-04-03T01:40:30.578 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700                           root   10u  netlink                            0t0         19238 KOBJECT_UEVENT
, 2022-04-03T01:40:30.579 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700                           root   11u  netlink                            0t0         19239 GENERIC

2022-04-03T14:13:12.868 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.622 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700   775 gmain               root  mem       REG                8,1          23         32624 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
, 2022-04-03T01:40:30.623 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700   775 gmain               root  mem       REG                8,1         368         32623 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
, 2022-04-03T01:40:30.623 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700   775 gmain               root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:12.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.656 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700   776 gdbus               root  mem       REG                8,1          54         28374 /usr/lib/locale/en_US.utf8/LC_NUMERIC
, 2022-04-03T01:40:30.657 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700   776 gdbus               root  mem       REG                8,1        3316         29664 /usr/lib/locale/en_US.utf8/LC_TIME
, 2022-04-03T01:40:30.657 INFO:teuthology.orchestra.run.smithi061.stdout:NetworkMa   700   776 gdbus               root  mem       REG                8,1         286         32625 /usr/lib/locale/en_US.utf8/LC_MONETARY

2022-04-03T14:13:12.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.691 INFO:teuthology.orchestra.run.smithi061.stdout:irqbalanc   706   752 gmain               root  mem       REG                8,1     1805368          9190 /usr/lib64/libunistring.so.2.1.0
, 2022-04-03T01:40:30.691 INFO:teuthology.orchestra.run.smithi061.stdout:irqbalanc   706   752 gmain               root  mem       REG                8,1      165624          9192 /usr/lib64/libidn2.so.0.3.6
, 2022-04-03T01:40:30.691 INFO:teuthology.orchestra.run.smithi061.stdout:irqbalanc   706   752 gmain               root  mem       REG                8,1     1246520          9254 /usr/lib64/libp11-kit.so.0.3.0

2022-04-03T14:13:12.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.725 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780                           root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:30.725 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780                           root  mem       REG                8,1     1187360          9165 /usr/lib64/libgcrypt.so.20.2.5
, 2022-04-03T01:40:30.726 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780                           root  DEL       REG                8,1                     10738 /usr/lib64/libmount.so.1.1.0

2022-04-03T14:13:12.872 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.759 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1010 tuned               root  mem       REG                8,1     1770248         11486 /usr/lib64/libgio-2.0.so.0.5600.4
, 2022-04-03T01:40:30.759 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1010 tuned               root  mem       REG                8,1     1167784         11487 /usr/lib64/libglib-2.0.so.0.5600.4
, 2022-04-03T01:40:30.760 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1010 tuned               root  mem       REG                8,1      347424         11489 /usr/lib64/libgobject-2.0.so.0.5600.4

2022-04-03T14:13:12.873 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.792 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1025 tuned               root  mem       REG                8,1     1541856          9167 /usr/lib64/libgmp.so.10.3.2
, 2022-04-03T01:40:30.792 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1025 tuned               root  mem       REG                8,1      197728          9361 /usr/lib64/libhogweed.so.4.5
, 2022-04-03T01:40:30.793 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1025 tuned               root  mem       REG                8,1      239360          9362 /usr/lib64/libnettle.so.6.5

2022-04-03T14:13:12.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.825 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1042 tuned               root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:30.826 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1042 tuned               root  txt       REG                8,1       11856         19642 /usr/libexec/platform-python3.6
, 2022-04-03T01:40:30.826 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1042 tuned               root  mem       REG               0,13                     11317 anon_inode:[perf_event] (stat: No such file or directory)

2022-04-03T14:13:12.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.858 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1042 tuned               root   10u  a_inode               0,13           0         11317 [perf_event]
, 2022-04-03T01:40:30.859 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1042 tuned               root   11u  a_inode               0,13           0         11317 [perf_event]
, 2022-04-03T01:40:30.859 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1042 tuned               root   12u  a_inode               0,13           0         11317 [perf_event]

2022-04-03T14:13:12.876 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.892 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1043 tuned               root  mem       REG                8,1          23         32624 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
, 2022-04-03T01:40:30.892 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1043 tuned               root  mem       REG                8,1         368         32623 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
, 2022-04-03T01:40:30.892 INFO:teuthology.orchestra.run.smithi061.stdout:tuned       780  1043 tuned               root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:12.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.925 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809                           root  mem       REG                8,1          34         23661 /usr/lib/locale/en_US.utf8/LC_PAPER
, 2022-04-03T01:40:30.925 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809                           root  mem       REG                8,1          77         28373 /usr/lib/locale/en_US.utf8/LC_NAME
, 2022-04-03T01:40:30.925 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809                           root  mem       REG                8,1         167         29712 /usr/lib/locale/en_US.utf8/LC_ADDRESS

2022-04-03T14:13:12.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.959 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809   811 gssproxy            root  mem       REG                8,1        3316         29664 /usr/lib/locale/en_US.utf8/LC_TIME
, 2022-04-03T01:40:30.959 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809   811 gssproxy            root  mem       REG                8,1         286         32625 /usr/lib/locale/en_US.utf8/LC_MONETARY
, 2022-04-03T01:40:30.959 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809   811 gssproxy            root  mem       REG                8,1          57         23647 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES

2022-04-03T14:13:12.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:30.992 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809   813 gssproxy            root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:30.992 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809   813 gssproxy            root  mem       REG                8,1      337024         28371 /usr/lib/locale/en_US.utf8/LC_CTYPE
, 2022-04-03T01:40:30.992 INFO:teuthology.orchestra.run.smithi061.stdout:gssproxy    809   813 gssproxy            root  mem       REG                8,1          54         28374 /usr/lib/locale/en_US.utf8/LC_NUMERIC

2022-04-03T14:13:12.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.025 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_be     830                           root  mem       REG                8,1        7648         43352 /usr/lib64/ldb/modules/ldb/ldb.so
, 2022-04-03T01:40:31.026 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_be     830                           root  mem       REG                8,1       16192          9159 /usr/lib64/libkeyutils.so.1.6
, 2022-04-03T01:40:31.026 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_be     830                           root  mem       REG                8,1       71480         10649 /usr/lib64/libkrb5support.so.0.1

2022-04-03T14:13:12.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.059 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_nss    862                           root  mem-w     REG                8,1     9253600         46453 /var/lib/sss/mc/passwd
, 2022-04-03T01:40:31.059 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_nss    862                           root  mem       REG                8,1     4136960          2044 /var/lib/sss/db/timestamps_implicit_files.ldb
, 2022-04-03T01:40:31.059 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_nss    862                           root  mem       REG                8,1     6086656          2048 /var/lib/sss/db/cache_implicit_files.ldb

2022-04-03T14:13:12.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.092 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_nss    862                           root   13u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:31.092 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_nss    862                           root   14u      REG                8,1     6086656          2048 /var/lib/sss/db/cache_implicit_files.ldb
, 2022-04-03T01:40:31.093 INFO:teuthology.orchestra.run.smithi061.stdout:sssd_nss    862                           root   15u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:12.881 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.125 INFO:teuthology.orchestra.run.smithi061.stdout:crond      1017                           root  mem       REG                8,1       46312         13315 /usr/lib64/libnss_sss.so.2
, 2022-04-03T01:40:31.125 INFO:teuthology.orchestra.run.smithi061.stdout:crond      1017                           root  mem       REG                8,1     2586930         23646 /usr/lib/locale/en_US.utf8/LC_COLLATE
, 2022-04-03T01:40:31.125 INFO:teuthology.orchestra.run.smithi061.stdout:crond      1017                           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so

2022-04-03T14:13:12.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.158 INFO:teuthology.orchestra.run.smithi061.stdout:systemd    1120                         ubuntu  mem       REG                8,1       66920          9158 /usr/lib64/libjson-c.so.4.0.0
, 2022-04-03T01:40:31.159 INFO:teuthology.orchestra.run.smithi061.stdout:systemd    1120                         ubuntu  mem       REG                8,1      371240         10922 /usr/lib64/libdevmapper.so.1.02
, 2022-04-03T01:40:31.159 INFO:teuthology.orchestra.run.smithi061.stdout:systemd    1120                         ubuntu  mem       REG                8,1       26704          9065 /usr/lib64/libattr.so.1.1.2448

2022-04-03T14:13:12.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.208 INFO:teuthology.orchestra.run.smithi061.stdout:(sd-pam)   1123                         ubuntu  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3
, 2022-04-03T01:40:31.208 INFO:teuthology.orchestra.run.smithi061.stdout:(sd-pam)   1123                         ubuntu  mem       REG                8,1      192024          8914 /usr/lib64/liblzma.so.5.2.4
, 2022-04-03T01:40:31.208 INFO:teuthology.orchestra.run.smithi061.stdout:(sd-pam)   1123                         ubuntu  mem       REG                8,1      165624          9192 /usr/lib64/libidn2.so.0.3.6

2022-04-03T14:13:12.884 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.241 INFO:teuthology.orchestra.run.smithi061.stdout:rsyslogd   1406  1408 in:imjour           root    4u     unix 0xffff8881038ea580         0t0         24331 type=DGRAM
, 2022-04-03T01:40:31.241 INFO:teuthology.orchestra.run.smithi061.stdout:rsyslogd   1406  1408 in:imjour           root    5r  a_inode               0,13           0         11317 inotify
, 2022-04-03T01:40:31.242 INFO:teuthology.orchestra.run.smithi061.stdout:rsyslogd   1406  1408 in:imjour           root    6w      REG                8,1        4715          1989 /var/log/cron

2022-04-03T14:13:12.886 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.274 INFO:teuthology.orchestra.run.smithi061.stdout:systemd-u 18761                           root    9r      REG                8,1     6940392         46446 /var/lib/sss/mc/group
, 2022-04-03T01:40:31.275 INFO:teuthology.orchestra.run.smithi061.stdout:systemd-u 18761                           root   11u     unix 0xffff88816e90da00         0t0        136314 type=DGRAM
, 2022-04-03T01:40:31.275 INFO:teuthology.orchestra.run.smithi061.stdout:systemd-u 18761                           root   12r  a_inode               0,13           0         11317 inotify

2022-04-03T14:13:12.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.308 INFO:teuthology.orchestra.run.smithi061.stdout:sshd      23493                           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:31.309 INFO:teuthology.orchestra.run.smithi061.stdout:sshd      23493                           root  mem       REG                8,1       24928          9106 /usr/lib64/libcap-ng.so.0.0.0
, 2022-04-03T01:40:31.309 INFO:teuthology.orchestra.run.smithi061.stdout:sshd      23493                           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so

2022-04-03T14:13:12.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.341 INFO:teuthology.orchestra.run.smithi061.stdout:sshd      23497                         ubuntu    5u     IPv4             154496         0t0           TCP smithi061.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:37836 (ESTABLISHED)
, 2022-04-03T01:40:31.342 INFO:teuthology.orchestra.run.smithi061.stdout:sshd      23497                         ubuntu    6u     unix 0xffff888170792d00         0t0        160261 type=STREAM
, 2022-04-03T01:40:31.342 INFO:teuthology.orchestra.run.smithi061.stdout:sshd      23497                         ubuntu    7r      REG                8,1    11567160         66009 /var/lib/sss/mc/initgroups (deleted)

2022-04-03T14:13:12.890 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.374 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-cras 23978                           root  mem       REG                8,1         286         32625 /usr/lib/locale/en_US.utf8/LC_MONETARY
, 2022-04-03T01:40:31.374 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-cras 23978                           root  mem       REG                8,1       26998         19541 /usr/lib64/gconv/gconv-modules.cache
, 2022-04-03T01:40:31.375 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-cras 23978                           root  mem       REG                8,1          57         23647 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES

2022-04-03T14:13:12.891 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.408 INFO:teuthology.orchestra.run.smithi061.stdout:xfs-log/d 30043                           root  cwd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:31.408 INFO:teuthology.orchestra.run.smithi061.stdout:xfs-log/d 30043                           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:31.408 INFO:teuthology.orchestra.run.smithi061.stdout:xfs-log/d 30043                           root  txt   unknown                                              /proc/30043/exe

2022-04-03T14:13:12.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.442 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36697                           root    9u     unix 0xffff888170795280         0t0        192628 type=DGRAM
, 2022-04-03T01:40:31.443 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36697                           root   10r     FIFO               0,12         0t0        192631 pipe
, 2022-04-03T01:40:31.443 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36697                           root   11w     FIFO               0,12         0t0        192631 pipe

2022-04-03T14:13:12.893 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.478 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771                           root  mem       REG                8,1       29256         38282 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.478 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771                           root  mem       REG                8,1       20688         40297 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.478 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771                           root  mem       REG                8,1       80112         38253 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.894 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.511 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36783 log                 root  mem       REG                8,1      120344         40302 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.511 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36783 log                 root  mem       REG                8,1       66064         38268 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.512 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36783 log                 root  mem       REG                8,1      131584         40296 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.894 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.545 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36785 msgr-work           root  mem       REG                8,1       66920          9158 /usr/lib64/libjson-c.so.4.0.0
, 2022-04-03T01:40:31.545 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36785 msgr-work           root  mem       REG                8,1      371240         10922 /usr/lib64/libdevmapper.so.1.02
, 2022-04-03T01:40:31.546 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36785 msgr-work           root  mem       REG                8,1      526384         10954 /usr/lib64/libcryptsetup.so.12.6.0

2022-04-03T14:13:12.895 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.579 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36785 msgr-work           root    5u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:31.579 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36785 msgr-work           root    6r     FIFO               0,12         0t0        193804 pipe
, 2022-04-03T01:40:31.579 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36785 msgr-work           root    7w     FIFO               0,12         0t0        193804 pipe

2022-04-03T14:13:12.896 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.613 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36786 msgr-work           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:31.613 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36786 msgr-work           root  mem       REG                8,1     2152080         24211 /usr/lib64/librados.so.2.0.0
, 2022-04-03T01:40:31.614 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36786 msgr-work           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so

2022-04-03T14:13:12.896 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.647 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36787 msgr-work           root  mem       REG                8,1       75328          8924 /usr/lib64/libbz2.so.1.0.6
, 2022-04-03T01:40:31.647 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36787 msgr-work           root  mem       REG                8,1       23032         40287 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.648 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36787 msgr-work           root  mem       REG                8,1       34392         38288 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.682 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36792 service             root  mem       REG                8,1       55488         38229 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.682 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36792 service             root  mem       REG                8,1       16888         38257 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.682 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36792 service             root  mem       REG                8,1       12288        525775 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.898 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.715 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36792 service             root   22w     FIFO               0,12         0t0        193825 pipe
, 2022-04-03T01:40:31.715 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36792 service             root   23u     IPv4             343513         0t0           TCP smithi061.front.sepia.ceph.com:60970->smithi067.front.sepia.ceph.com:6832 (ESTABLISHED)
, 2022-04-03T01:40:31.715 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36792 service             root   25u     IPv4             244141         0t0           TCP smithi061.front.sepia.ceph.com:47526->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)

2022-04-03T14:13:12.898 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.750 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36793 admin_soc           root  mem       REG                8,1     3289680         10304 /usr/lib64/libpython3.6m.so.1.0
, 2022-04-03T01:40:31.750 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36793 admin_soc           root  mem       REG                8,1     1151744          9009 /usr/lib64/libsqlite3.so.0.8.6
, 2022-04-03T01:40:31.751 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36793 admin_soc           root  mem       REG                8,1      536160         24237 /usr/lib64/libcephsqlite.so

2022-04-03T14:13:12.899 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.784 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36794 ceph_time           root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:31.784 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36794 ceph_time           root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:31.785 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36794 ceph_time           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0

2022-04-03T14:13:12.900 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.822 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36795 fn_anonym           root  mem       REG                8,1     2543752         24224 /usr/lib64/libcephfs.so.2.0.0
, 2022-04-03T01:40:31.822 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36795 fn_anonym           root  mem       REG                8,1      574864         31940 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.823 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36795 fn_anonym           root  mem       REG                8,1       20528         40298 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.900 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.855 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36796 signal_ha           root  mem       REG                8,1       66904         38279 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.856 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36796 signal_ha           root  mem       REG                8,1       39464         38241 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.856 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36796 signal_ha           root  mem       REG                8,1      930848        525286 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.901 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.889 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36796 signal_ha           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:31.890 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36796 signal_ha           root   12r     FIFO               0,12         0t0        193806 pipe
, 2022-04-03T01:40:31.890 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36796 signal_ha           root   13w     FIFO               0,12         0t0        193806 pipe

2022-04-03T14:13:12.901 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.923 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36797 mgrsb-fin           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:31.923 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36797 mgrsb-fin           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:31.923 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36797 mgrsb-fin           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:12.902 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.957 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36798 ms_dispat           root  mem       REG                8,1      110024         40301 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.958 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36798 ms_dispat           root  mem       REG                8,1       92992         40300 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:31.958 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36798 ms_dispat           root  mem       REG                8,1      115208         38245 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.903 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:31.991 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36799 ms_local            root  mem       REG                8,1    38081912         30134 /usr/lib64/libopenblas-r0.3.15.so
, 2022-04-03T01:40:31.991 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36799 ms_local            root  mem       REG                8,1    39605944         30137 /usr/lib64/libopenblasp-r0.3.15.so
, 2022-04-03T01:40:31.991 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36799 ms_local            root  mem       REG                8,1     1736392        524957 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.903 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.023 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36800 io_contex           root  mem       REG                8,1       29992         40311 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.024 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36800 io_contex           root  mem       REG                8,1      239360          9362 /usr/lib64/libnettle.so.6.5
, 2022-04-03T01:40:32.024 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36800 io_contex           root  mem       REG                8,1       76048         38248 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.904 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.055 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36800 io_contex           root    0r      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:32.056 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36800 io_contex           root    1w     FIFO               0,12         0t0        193738 pipe
, 2022-04-03T01:40:32.056 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36800 io_contex           root    2w     FIFO               0,12         0t0        193739 pipe

2022-04-03T14:13:12.905 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.090 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36801 io_contex           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:32.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36801 io_contex           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:32.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36801 io_contex           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0

2022-04-03T14:13:12.905 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.127 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36802 safe_time           root  mem       REG                8,1      615712         15747 /usr/lib64/libssl.so.1.1.1k
, 2022-04-03T01:40:32.127 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36802 safe_time           root  mem       REG                8,1       47920         38250 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.128 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36802 safe_time           root  mem       REG                8,1       13208         40306 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.906 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.163 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36803 safe_time           root  mem       REG                8,1       20960        525208 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.164 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36803 safe_time           root  mem       REG                8,1     1474656        525018 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.164 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36803 safe_time           root  mem       REG                8,1      824512        657805 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so

2022-04-03T14:13:12.907 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.197 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36803 safe_time           root   17w     FIFO               0,12         0t0        193819 pipe
, 2022-04-03T01:40:32.197 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36803 safe_time           root   18u     unix 0xffff888107006180         0t0        193820 /var/run/ceph/ceph-mgr.y.asok type=STREAM
, 2022-04-03T01:40:32.197 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36803 safe_time           root   19r     FIFO               0,12         0t0        193824 pipe

2022-04-03T14:13:12.907 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.231 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36804 safe_time           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:32.231 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36804 safe_time           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:32.232 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36804 safe_time           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3

2022-04-03T14:13:12.908 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.265 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36805 fn_anonym           root  mem       REG                8,1       20688         40297 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.265 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36805 fn_anonym           root  mem       REG                8,1       80112         38253 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.266 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36805 fn_anonym           root  mem       REG                8,1       11592         38258 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:12.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.300 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36806 flusher             root  mem       REG                8,1       66064         38268 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.300 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36806 flusher             root  mem       REG                8,1      131584         40296 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so
, 2022-04-03T01:40:32.301 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36806 flusher             root  mem       REG                8,1      134584          9638 /usr/lib64/libyaml-0.so.2.0.5

2022-04-03T14:13:12.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.334 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36807 safe_time           root  mem       REG                8,1      371240         10922 /usr/lib64/libdevmapper.so.1.02
, 2022-04-03T01:40:32.335 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36807 safe_time           root  mem       REG                8,1      526384         10954 /usr/lib64/libcryptsetup.so.12.6.0
, 2022-04-03T01:40:32.335 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36807 safe_time           root  mem       REG                8,1    12743336         24219 /usr/lib64/librbd.so.1.17.0

2022-04-03T14:13:12.910 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.369 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36807 safe_time           root    6r     FIFO               0,12         0t0        193804 pipe
, 2022-04-03T01:40:32.369 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36807 safe_time           root    7w     FIFO               0,12         0t0        193804 pipe
, 2022-04-03T01:40:32.369 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mgr  36771 36807 safe_time           root    8u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:12.911 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.404 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36938                           root   11w     FIFO               0,12         0t0        195916 pipe
, 2022-04-03T01:40:32.404 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36941                           root  cwd       DIR                8,1        4096          4714 /home/ubuntu
, 2022-04-03T01:40:32.404 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36941                           root  rtd       DIR                8,1        4096             2 /

2022-04-03T14:13:12.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.439 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36946                           root  mem       REG                8,1       11776         31857 /usr/lib64/security/pam_localuser.so
, 2022-04-03T01:40:32.439 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36946                           root  mem       REG                8,1       11856         31875 /usr/lib64/security/pam_usertype.so
, 2022-04-03T01:40:32.439 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36946                           root  mem       REG                8,1       11752         31851 /usr/lib64/security/pam_faildelay.so

2022-04-03T14:13:12.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.473 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36953                           root  mem       REG                8,1      971456         10648 /usr/lib64/libkrb5.so.3.3
, 2022-04-03T01:40:32.474 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36953                           root  mem       REG                8,1      355920         10640 /usr/lib64/libgssapi_krb5.so.2.2
, 2022-04-03T01:40:32.474 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      36953                           root  mem       REG                8,1      136040          8933 /usr/lib64/libcrypt.so.1.1.0

2022-04-03T14:13:12.913 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.508 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 37035                           root  mem       REG                8,1      337024         28371 /usr/lib/locale/en_US.utf8/LC_CTYPE
, 2022-04-03T01:40:32.508 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 37035                           root  mem       REG                8,1          54         28374 /usr/lib/locale/en_US.utf8/LC_NUMERIC
, 2022-04-03T01:40:32.508 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 37035                           root  mem       REG                8,1        3316         29664 /usr/lib/locale/en_US.utf8/LC_TIME

2022-04-03T14:13:12.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.542 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051                           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0
, 2022-04-03T01:40:32.542 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051                           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0
, 2022-04-03T01:40:32.543 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051                           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0

2022-04-03T14:13:12.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.576 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051                           root   61r     FIFO               0,12         0t0        201957 pipe
, 2022-04-03T01:40:32.576 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051                           root   62w     FIFO               0,12         0t0        201957 pipe
, 2022-04-03T01:40:32.576 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051                           root   63u     IPv4             197101         0t0           TCP smithi061.front.sepia.ceph.com:52980->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)

2022-04-03T14:13:12.915 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.609 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37070 log                 root    7w     FIFO               0,12         0t0        194055 pipe
, 2022-04-03T01:40:32.610 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37070 log                 root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:32.610 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37070 log                 root    9r     FIFO               0,12         0t0        194056 pipe

2022-04-03T14:13:12.915 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.643 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37072 msgr-work           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0
, 2022-04-03T01:40:32.644 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37072 msgr-work           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:32.644 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37072 msgr-work           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0

2022-04-03T14:13:12.916 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.678 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37072 msgr-work           root   56u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.678 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37072 msgr-work           root   57r     FIFO               0,12         0t0        201956 pipe
, 2022-04-03T01:40:32.679 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37072 msgr-work           root   58u     IPv4             271593         0t0           TCP smithi061.front.sepia.ceph.com:6810->smithi067.front.sepia.ceph.com:49784 (ESTABLISHED)

2022-04-03T14:13:12.917 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.713 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37073 msgr-work           root    2w     FIFO               0,12         0t0        192771 pipe
, 2022-04-03T01:40:32.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37073 msgr-work           root    3r     FIFO               0,12         0t0        194054 pipe
, 2022-04-03T01:40:32.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37073 msgr-work           root    4w     FIFO               0,12         0t0        194054 pipe

2022-04-03T14:13:12.917 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.747 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37074 msgr-work           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:32.748 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37074 msgr-work           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:32.748 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37074 msgr-work           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0

2022-04-03T14:13:12.918 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.781 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37074 msgr-work           root   51u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.781 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37074 msgr-work           root   52u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.782 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37074 msgr-work           root   53u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.815 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37093 service             root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:32.816 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37093 service             root  DEL       REG               0,17                    200078 /[aio]
, 2022-04-03T01:40:32.816 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37093 service             root  DEL       REG               0,17                    196048 /[aio]

2022-04-03T14:13:12.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.850 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37095 admin_soc           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0
, 2022-04-03T01:40:32.850 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37095 admin_soc           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0
, 2022-04-03T01:40:32.851 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37095 admin_soc           root  mem       REG                8,1     1338272        789117 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0

2022-04-03T14:13:12.920 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.884 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37095 admin_soc           root   46u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.884 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37095 admin_soc           root   47u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.884 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37095 admin_soc           root   48u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.918 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37143 signal_ha           root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3
, 2022-04-03T01:40:32.918 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37143 signal_ha           root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:32.919 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37143 signal_ha           root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1

2022-04-03T14:13:12.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37143 signal_ha           root   99u     IPv4             220149         0t0           TCP smithi061.front.sepia.ceph.com:47460->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)
, 2022-04-03T01:40:32.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37143 signal_ha           root  103u     IPv4             242196         0t0           TCP smithi061.front.sepia.ceph.com:47512->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:32.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37144 io_contex           root  cwd       DIR                8,1        4096          4714 /home/ubuntu

2022-04-03T14:13:12.922 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:32.987 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37144 io_contex           root   41u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.987 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37144 io_contex           root   42u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:32.988 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37144 io_contex           root   43u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.021 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37145 io_contex           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k
, 2022-04-03T01:40:33.022 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37145 io_contex           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:33.022 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37145 io_contex           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so

2022-04-03T14:13:12.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.056 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37145 io_contex           root   89u     IPv4             197134         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi061.front.sepia.ceph.com:49372 (ESTABLISHED)
, 2022-04-03T01:40:33.056 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37145 io_contex           root   90u     IPv4             220148         0t0           TCP smithi061.front.sepia.ceph.com:59032->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)
, 2022-04-03T01:40:33.056 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37145 io_contex           root   91u     IPv4             197141         0t0           TCP smithi061.front.sepia.ceph.com:45840->smithi061.front.sepia.ceph.com:6821 (ESTABLISHED)

2022-04-03T14:13:12.924 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.090 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37164 OpHistory           root   36u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37164 OpHistory           root   37u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37164 OpHistory           root   38u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.125 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37165 ceph_time           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:33.125 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37165 ceph_time           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:33.126 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37165 ceph_time           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:12.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.159 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37165 ceph_time           root   84u     IPv4             197121         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35834 (ESTABLISHED)
, 2022-04-03T01:40:33.160 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37165 ceph_time           root   85u     IPv4             197122         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35856 (ESTABLISHED)
, 2022-04-03T01:40:33.160 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37165 ceph_time           root   86u     IPv4             197125         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi061.front.sepia.ceph.com:49370 (ESTABLISHED)

2022-04-03T14:13:12.926 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37199 ceph-osd            root   31r      DIR              253,2         278            64 /var/lib/ceph/osd/ceph-2
, 2022-04-03T01:40:33.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37199 ceph-osd            root   32uW     REG              253,2          37            69 /var/lib/ceph/osd/ceph-2/fsid
, 2022-04-03T01:40:33.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37199 ceph-osd            root   33u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.927 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.228 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37236 ceph-osd            root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1
, 2022-04-03T01:40:33.228 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37236 ceph-osd            root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:33.228 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37236 ceph-osd            root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so

2022-04-03T14:13:12.927 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.262 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37236 ceph-osd            root   79u     IPv4             197112         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi067.front.sepia.ceph.com:55736 (ESTABLISHED)
, 2022-04-03T01:40:33.262 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37236 ceph-osd            root   80u     IPv4             197113         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi061.front.sepia.ceph.com:49368 (ESTABLISHED)
, 2022-04-03T01:40:33.262 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37236 ceph-osd            root   81u     IPv4             197116         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi067.front.sepia.ceph.com:55776 (ESTABLISHED)

2022-04-03T14:13:12.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37250 ceph-osd            root   26u     IPv4             194231         0t0           TCP *:6815 (LISTEN)
, 2022-04-03T01:40:33.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37250 ceph-osd            root   27r     FIFO               0,12         0t0        195971 pipe
, 2022-04-03T01:40:33.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37250 ceph-osd            root   28w     FIFO               0,12         0t0        195971 pipe

2022-04-03T14:13:12.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.330 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37263 ceph-osd            root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:33.330 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37263 ceph-osd            root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1
, 2022-04-03T01:40:33.331 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37263 ceph-osd            root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0

2022-04-03T14:13:12.929 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.363 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37263 ceph-osd            root   74u     IPv4             197105         0t0           TCP smithi061.front.sepia.ceph.com:33366->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:33.364 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37263 ceph-osd            root   75u     IPv4             197106         0t0           TCP smithi061.front.sepia.ceph.com:51300->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:33.364 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37263 ceph-osd            root   76u     IPv4             197107         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35788 (ESTABLISHED)

2022-04-03T14:13:12.930 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.396 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37276 ceph-osd            root   21u     IPv4             194161         0t0           TCP *:6810 (LISTEN)
, 2022-04-03T01:40:33.397 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37276 ceph-osd            root   22u     IPv4             194173         0t0           TCP *:6811 (LISTEN)
, 2022-04-03T01:40:33.397 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37276 ceph-osd            root   23u     IPv4             194186         0t0           TCP *:6812 (LISTEN)

2022-04-03T14:13:12.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.430 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37290 ceph-osd            root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0
, 2022-04-03T01:40:33.430 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37290 ceph-osd            root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:33.430 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37290 ceph-osd            root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0

2022-04-03T14:13:12.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.462 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37290 ceph-osd            root   69u     IPv4             201178         0t0           TCP smithi061.front.sepia.ceph.com:32954->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:33.463 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37290 ceph-osd            root   70u     IPv4             199517         0t0           TCP smithi061.front.sepia.ceph.com:60140->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)
, 2022-04-03T01:40:33.463 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37290 ceph-osd            root   71u     IPv4             201179         0t0           TCP smithi061.front.sepia.ceph.com:36282->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)

2022-04-03T14:13:12.932 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.495 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37305 ceph-osd            root   16r     FIFO               0,12         0t0        195963 pipe
, 2022-04-03T01:40:33.495 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37305 ceph-osd            root   17w     FIFO               0,12         0t0        195963 pipe
, 2022-04-03T01:40:33.495 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37305 ceph-osd            root   18u     unix 0xffff888170534b00         0t0        195964 /var/run/ceph/ceph-osd.2.asok type=STREAM

2022-04-03T14:13:12.932 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.528 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37317 ceph-osd            root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:33.529 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37317 ceph-osd            root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0
, 2022-04-03T01:40:33.529 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37317 ceph-osd            root  mem       REG                8,1       57160        789048 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0

2022-04-03T14:13:12.933 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.562 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37317 ceph-osd            root   64u     IPv4             197102         0t0           TCP smithi061.front.sepia.ceph.com:36810->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:33.562 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37317 ceph-osd            root   65u     IPv4             197103         0t0           TCP smithi061.front.sepia.ceph.com:44910->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:33.562 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37317 ceph-osd            root   66u     IPv4             199513         0t0           TCP smithi061.front.sepia.ceph.com:35636->smithi061.front.sepia.ceph.com:6806 (ESTABLISHED)

2022-04-03T14:13:12.934 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.595 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37318 safe_time           root   10w     FIFO               0,12         0t0        194056 pipe
, 2022-04-03T01:40:33.596 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37318 safe_time           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:33.596 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37318 safe_time           root   12r     FIFO               0,12         0t0        194057 pipe

2022-04-03T14:13:12.934 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.629 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37319 safe_time           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:33.629 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37319 safe_time           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:33.629 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37319 safe_time           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0

2022-04-03T14:13:12.935 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.662 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37319 safe_time           root   59w     FIFO               0,12         0t0        201956 pipe
, 2022-04-03T01:40:33.662 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37319 safe_time           root   60u     IPv4             199512         0t0           TCP smithi061.front.sepia.ceph.com:53398->smithi061.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:33.662 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37319 safe_time           root   61r     FIFO               0,12         0t0        201957 pipe

2022-04-03T14:13:12.936 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.695 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37320 safe_time           root    5u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:33.696 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37320 safe_time           root    6r     FIFO               0,12         0t0        194055 pipe
, 2022-04-03T01:40:33.696 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37320 safe_time           root    7w     FIFO               0,12         0t0        194055 pipe

2022-04-03T14:13:12.936 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.729 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37321 safe_time           root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0
, 2022-04-03T01:40:33.729 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37321 safe_time           root  mem       REG                8,1       70768        789090 /usr/lib64/rados-classes/libcls_hello.so.1.0.0
, 2022-04-03T01:40:33.729 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37321 safe_time           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0

2022-04-03T14:13:12.937 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.762 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37321 safe_time           root   54u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.762 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37321 safe_time           root   55u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.762 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37321 safe_time           root   56u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.938 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.795 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37323 fn_anonym           root    0r      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:33.795 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37323 fn_anonym           root    1w     FIFO               0,12         0t0        192770 pipe
, 2022-04-03T01:40:33.796 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37323 fn_anonym           root    2w     FIFO               0,12         0t0        192771 pipe

2022-04-03T14:13:12.938 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.834 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37328 rocksdb:l           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0
, 2022-04-03T01:40:33.834 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37328 rocksdb:l           root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:33.834 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37328 rocksdb:l           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0

2022-04-03T14:13:12.939 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37328 rocksdb:l           root   49u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.873 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37328 rocksdb:l           root   50u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.873 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37328 rocksdb:l           root   51u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.940 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.908 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37329 rocksdb:l           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7
, 2022-04-03T01:40:33.908 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37329 rocksdb:l           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:33.909 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37329 rocksdb:l           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so

2022-04-03T14:13:12.940 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.946 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37330 rocksdb:h           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:33.947 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37330 rocksdb:h           root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:33.947 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37330 rocksdb:h           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0

2022-04-03T14:13:12.941 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:33.979 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37330 rocksdb:h           root   44u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.979 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37330 rocksdb:h           root   45u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:33.979 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 37330 rocksdb:h           root   46u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.942 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.012 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38826 bstore_ai           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:34.012 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38826 bstore_ai           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:34.013 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38826 bstore_ai           root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3

2022-04-03T14:13:12.942 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.047 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38826 bstore_ai           root   97u     IPv4             199551         0t0           TCP smithi061.front.sepia.ceph.com:6810->smithi067.front.sepia.ceph.com:48536 (ESTABLISHED)
, 2022-04-03T01:40:34.047 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38826 bstore_ai           root   98u     IPv4             196267         0t0           TCP smithi061.front.sepia.ceph.com:40470->smithi061.front.sepia.ceph.com:6802 (ESTABLISHED)
, 2022-04-03T01:40:34.047 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38826 bstore_ai           root   99u     IPv4             220149         0t0           TCP smithi061.front.sepia.ceph.com:47460->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)

2022-04-03T14:13:12.943 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.080 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38827 bstore_di           root   39u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.080 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38827 bstore_di           root   40u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.080 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 38827 bstore_di           root   41u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.944 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.112 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39010 bstore_ai           root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11
, 2022-04-03T01:40:34.113 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39010 bstore_ai           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so
, 2022-04-03T01:40:34.113 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39010 bstore_ai           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k

2022-04-03T14:13:12.944 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.144 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39010 bstore_ai           root   87u     IPv4             197128         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi061.front.sepia.ceph.com:44936 (ESTABLISHED)
, 2022-04-03T01:40:34.145 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39010 bstore_ai           root   88u     IPv4             197131         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi061.front.sepia.ceph.com:44938 (ESTABLISHED)
, 2022-04-03T01:40:34.145 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39010 bstore_ai           root   89u     IPv4             197134         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi061.front.sepia.ceph.com:49372 (ESTABLISHED)

2022-04-03T14:13:12.945 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.179 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39011 bstore_di           root   34u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.180 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39011 bstore_di           root   35u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.180 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39011 bstore_di           root   36u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.945 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.214 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39486 ceph-osd            root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:34.214 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39486 ceph-osd            root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so
, 2022-04-03T01:40:34.214 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39486 ceph-osd            root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25

2022-04-03T14:13:12.946 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.247 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39486 ceph-osd            root   82u     IPv4             197117         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi067.front.sepia.ceph.com:55778 (ESTABLISHED)
, 2022-04-03T01:40:34.247 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39486 ceph-osd            root   83u     IPv4             197120         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi061.front.sepia.ceph.com:44934 (ESTABLISHED)
, 2022-04-03T01:40:34.247 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39486 ceph-osd            root   84u     IPv4             197121         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35834 (ESTABLISHED)

2022-04-03T14:13:12.947 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.280 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39502 cfin                root   29r     FIFO               0,12         0t0        195972 pipe
, 2022-04-03T01:40:34.280 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39502 cfin                root   30w     FIFO               0,12         0t0        195972 pipe
, 2022-04-03T01:40:34.281 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39502 cfin                root   31r      DIR              253,2         278            64 /var/lib/ceph/osd/ceph-2

2022-04-03T14:13:12.948 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.313 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39503 bstore_kv           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:34.313 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39503 bstore_kv           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:34.314 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39503 bstore_kv           root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1

2022-04-03T14:13:12.948 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.347 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39503 bstore_kv           root   77u     IPv4             197108         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35804 (ESTABLISHED)
, 2022-04-03T01:40:34.347 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39503 bstore_kv           root   78u     IPv4             197111         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi067.front.sepia.ceph.com:55718 (ESTABLISHED)
, 2022-04-03T01:40:34.348 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39503 bstore_kv           root   79u     IPv4             197112         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi067.front.sepia.ceph.com:55736 (ESTABLISHED)

2022-04-03T14:13:12.949 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.380 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39505 bstore_kv           root   24u     IPv4             194200         0t0           TCP *:6813 (LISTEN)
, 2022-04-03T01:40:34.380 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39505 bstore_kv           root   25u     IPv4             194215         0t0           TCP *:6814 (LISTEN)
, 2022-04-03T01:40:34.381 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39505 bstore_kv           root   26u     IPv4             194231         0t0           TCP *:6815 (LISTEN)

2022-04-03T14:13:12.949 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.413 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39509 bstore_me           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0
, 2022-04-03T01:40:34.413 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39509 bstore_me           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:34.413 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39509 bstore_me           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so

2022-04-03T14:13:12.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.446 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39509 bstore_me           root   72u     IPv4             199518         0t0           TCP smithi061.front.sepia.ceph.com:34844->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)
, 2022-04-03T01:40:34.446 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39509 bstore_me           root   73u     IPv4             201180         0t0           TCP smithi061.front.sepia.ceph.com:39966->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)
, 2022-04-03T01:40:34.446 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39509 bstore_me           root   74u     IPv4             197105         0t0           TCP smithi061.front.sepia.ceph.com:33366->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)

2022-04-03T14:13:12.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.479 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39652 ms_dispat           root   19u     IPv4             194140         0t0           TCP *:6808 (LISTEN)
, 2022-04-03T01:40:34.479 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39652 ms_dispat           root   20u     IPv4             194150         0t0           TCP *:6809 (LISTEN)
, 2022-04-03T01:40:34.480 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39652 ms_dispat           root   21u     IPv4             194161         0t0           TCP *:6810 (LISTEN)

2022-04-03T14:13:12.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.512 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39654 ms_local            root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:34.513 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39654 ms_local            root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:34.513 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39654 ms_local            root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:12.952 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.547 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39654 ms_local            root   67u     IPv4             197104         0t0           TCP smithi061.front.sepia.ceph.com:34572->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:34.547 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39654 ms_local            root   68u     IPv4             199516         0t0           TCP smithi061.front.sepia.ceph.com:40144->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:34.548 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39654 ms_local            root   69u     IPv4             201178         0t0           TCP smithi061.front.sepia.ceph.com:32954->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)

2022-04-03T14:13:12.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.579 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39656 safe_time           root   13w     FIFO               0,12         0t0        194057 pipe
, 2022-04-03T01:40:34.579 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39656 safe_time           root   14w      REG                8,1  3960831205        790594 /var/log/ceph/ceph-osd.2.log
, 2022-04-03T01:40:34.579 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39656 safe_time           root   16r     FIFO               0,12         0t0        195963 pipe

2022-04-03T14:13:12.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.610 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39658 safe_time           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0
, 2022-04-03T01:40:34.610 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39658 safe_time           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:34.611 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39658 safe_time           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0

2022-04-03T14:13:12.954 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.643 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39658 safe_time           root   62w     FIFO               0,12         0t0        201957 pipe
, 2022-04-03T01:40:34.643 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39658 safe_time           root   63u     IPv4             197101         0t0           TCP smithi061.front.sepia.ceph.com:52980->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:34.644 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39658 safe_time           root   64u     IPv4             197102         0t0           TCP smithi061.front.sepia.ceph.com:36810->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)

2022-04-03T14:13:12.955 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.677 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39660 ms_dispat           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:34.677 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39660 ms_dispat           root    9r     FIFO               0,12         0t0        194056 pipe
, 2022-04-03T01:40:34.678 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39660 ms_dispat           root   10w     FIFO               0,12         0t0        194056 pipe

2022-04-03T14:13:12.955 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.710 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39663 ms_local            root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:34.711 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39663 ms_local            root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:34.711 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39663 ms_local            root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0

2022-04-03T14:13:12.956 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.744 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39663 ms_local            root   56u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.744 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39663 ms_local            root   57r     FIFO               0,12         0t0        201956 pipe
, 2022-04-03T01:40:34.745 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39663 ms_local            root   58u     IPv4             271593         0t0           TCP smithi061.front.sepia.ceph.com:6810->smithi067.front.sepia.ceph.com:49784 (ESTABLISHED)

2022-04-03T14:13:12.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.778 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39665 ms_dispat           root    2w     FIFO               0,12         0t0        192771 pipe
, 2022-04-03T01:40:34.778 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39665 ms_dispat           root    3r     FIFO               0,12         0t0        194054 pipe
, 2022-04-03T01:40:34.779 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39665 ms_dispat           root    4w     FIFO               0,12         0t0        194054 pipe

2022-04-03T14:13:12.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.812 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39667 ms_local            root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:34.812 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39667 ms_local            root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:34.812 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39667 ms_local            root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3

2022-04-03T14:13:12.958 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.845 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39667 ms_local            root   50u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.845 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39667 ms_local            root   51u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.845 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39667 ms_local            root   52u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.959 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.878 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39668 ms_dispat           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7
, 2022-04-03T01:40:34.879 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39668 ms_dispat           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:34.879 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39668 ms_dispat           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so

2022-04-03T14:13:12.959 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.912 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39670 ms_local            root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:34.913 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39670 ms_local            root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:34.913 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39670 ms_local            root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0

2022-04-03T14:13:12.960 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.946 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39670 ms_local            root   44u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.947 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39670 ms_local            root   45u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:34.947 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39670 ms_local            root   46u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.961 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:34.982 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39674 ms_dispat           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so
, 2022-04-03T01:40:34.982 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39674 ms_dispat           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k
, 2022-04-03T01:40:34.982 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39674 ms_dispat           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so

2022-04-03T14:13:12.961 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.015 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39674 ms_dispat           root   88u     IPv4             197131         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi061.front.sepia.ceph.com:44938 (ESTABLISHED)
, 2022-04-03T01:40:35.015 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39674 ms_dispat           root   89u     IPv4             197134         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi061.front.sepia.ceph.com:49372 (ESTABLISHED)
, 2022-04-03T01:40:35.015 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39674 ms_dispat           root   90u     IPv4             220148         0t0           TCP smithi061.front.sepia.ceph.com:59032->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)

2022-04-03T14:13:12.962 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.047 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39676 ms_local            root   35u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.047 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39676 ms_local            root   36u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.048 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39676 ms_local            root   37u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.963 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.080 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39678 ms_dispat           root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so
, 2022-04-03T01:40:35.080 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39678 ms_dispat           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:35.081 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39678 ms_dispat           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2

2022-04-03T14:13:12.963 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.113 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39678 ms_dispat           root   83u     IPv4             197120         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi061.front.sepia.ceph.com:44934 (ESTABLISHED)
, 2022-04-03T01:40:35.113 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39678 ms_dispat           root   84u     IPv4             197121         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35834 (ESTABLISHED)
, 2022-04-03T01:40:35.113 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39678 ms_dispat           root   85u     IPv4             197122         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35856 (ESTABLISHED)

2022-04-03T14:13:12.964 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.145 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39680 ms_local            root   30w     FIFO               0,12         0t0        195972 pipe
, 2022-04-03T01:40:35.145 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39680 ms_local            root   31r      DIR              253,2         278            64 /var/lib/ceph/osd/ceph-2
, 2022-04-03T01:40:35.146 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39680 ms_local            root   32uW     REG              253,2          37            69 /var/lib/ceph/osd/ceph-2/fsid

2022-04-03T14:13:12.965 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.179 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39682 ms_dispat           root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1
, 2022-04-03T01:40:35.179 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39682 ms_dispat           root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0
, 2022-04-03T01:40:35.179 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39682 ms_dispat           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0

2022-04-03T14:13:12.965 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.211 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39682 ms_dispat           root   75u     IPv4             197106         0t0           TCP smithi061.front.sepia.ceph.com:51300->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:35.211 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39682 ms_dispat           root   76u     IPv4             197107         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35788 (ESTABLISHED)
, 2022-04-03T01:40:35.212 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39682 ms_dispat           root   77u     IPv4             197108         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35804 (ESTABLISHED)

2022-04-03T14:13:12.966 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.245 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39684 ms_local            root   22u     IPv4             194173         0t0           TCP *:6811 (LISTEN)
, 2022-04-03T01:40:35.245 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39684 ms_local            root   23u     IPv4             194186         0t0           TCP *:6812 (LISTEN)
, 2022-04-03T01:40:35.245 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39684 ms_local            root   24u     IPv4             194200         0t0           TCP *:6813 (LISTEN)

2022-04-03T14:13:12.967 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.279 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39686 fn_anonym           root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:35.280 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39686 fn_anonym           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:35.280 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39686 fn_anonym           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:12.967 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.313 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39686 fn_anonym           root   67u     IPv4             197104         0t0           TCP smithi061.front.sepia.ceph.com:34572->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:35.313 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39686 fn_anonym           root   68u     IPv4             199516         0t0           TCP smithi061.front.sepia.ceph.com:40144->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:35.313 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39686 fn_anonym           root   69u     IPv4             201178         0t0           TCP smithi061.front.sepia.ceph.com:32954->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)

2022-04-03T14:13:12.968 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.347 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39688 finisher            root   10w     FIFO               0,12         0t0        194056 pipe
, 2022-04-03T01:40:35.348 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39688 finisher            root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:35.348 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39688 finisher            root   12r     FIFO               0,12         0t0        194057 pipe

2022-04-03T14:13:12.969 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.381 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39690 safe_time           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:35.381 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39690 safe_time           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:35.382 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39690 safe_time           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0

2022-04-03T14:13:12.969 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.422 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39690 safe_time           root   57r     FIFO               0,12         0t0        201956 pipe
, 2022-04-03T01:40:35.422 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39690 safe_time           root   58u     IPv4             271593         0t0           TCP smithi061.front.sepia.ceph.com:6810->smithi067.front.sepia.ceph.com:49784 (ESTABLISHED)
, 2022-04-03T01:40:35.422 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39690 safe_time           root   59w     FIFO               0,12         0t0        201956 pipe

2022-04-03T14:13:12.970 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.459 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39691 safe_time           root    3r     FIFO               0,12         0t0        194054 pipe
, 2022-04-03T01:40:35.459 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39691 safe_time           root    4w     FIFO               0,12         0t0        194054 pipe
, 2022-04-03T01:40:35.460 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39691 safe_time           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:12.971 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.495 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39693 ceph-osd            root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:35.496 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39693 ceph-osd            root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0
, 2022-04-03T01:40:35.496 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39693 ceph-osd            root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0

2022-04-03T14:13:12.971 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.534 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39693 ceph-osd            root   52u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.534 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39693 ceph-osd            root   53u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.534 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39693 ceph-osd            root   54u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.972 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.566 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39695 osd_srv_a           root  DEL       REG               0,17                    200078 /[aio]
, 2022-04-03T01:40:35.567 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39695 osd_srv_a           root  DEL       REG               0,17                    196048 /[aio]
, 2022-04-03T01:40:35.567 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39695 osd_srv_a           root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:12.973 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.599 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39697 tp_osd_tp           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0
, 2022-04-03T01:40:35.599 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39697 tp_osd_tp           root  mem       REG                8,1     1338272        789117 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0
, 2022-04-03T01:40:35.600 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39697 tp_osd_tp           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0

2022-04-03T14:13:12.973 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.632 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39697 tp_osd_tp           root   47u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.632 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39697 tp_osd_tp           root   48u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.633 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39697 tp_osd_tp           root   49u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.974 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.666 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39698 tp_osd_tp           root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:35.666 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39698 tp_osd_tp           root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1
, 2022-04-03T01:40:35.666 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39698 tp_osd_tp           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7

2022-04-03T14:13:12.975 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.700 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39698 tp_osd_tp           root  103u     IPv4             242196         0t0           TCP smithi061.front.sepia.ceph.com:47512->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:35.700 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39700 tp_osd_tp           root  cwd       DIR                8,1        4096          4714 /home/ubuntu
, 2022-04-03T01:40:35.701 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39700 tp_osd_tp           root  rtd       DIR                8,1        4096             2 /

2022-04-03T14:13:12.975 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.733 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39700 tp_osd_tp           root   42u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.734 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39700 tp_osd_tp           root   43u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.734 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39700 tp_osd_tp           root   44u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.976 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.767 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39703 tp_osd_tp           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:35.767 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39703 tp_osd_tp           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:35.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39703 tp_osd_tp           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3

2022-04-03T14:13:12.977 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.802 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39703 tp_osd_tp           root   90u     IPv4             220148         0t0           TCP smithi061.front.sepia.ceph.com:59032->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)
, 2022-04-03T01:40:35.802 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39703 tp_osd_tp           root   91u     IPv4             197141         0t0           TCP smithi061.front.sepia.ceph.com:45840->smithi061.front.sepia.ceph.com:6821 (ESTABLISHED)
, 2022-04-03T01:40:35.802 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39703 tp_osd_tp           root   97u     IPv4             199551         0t0           TCP smithi061.front.sepia.ceph.com:6810->smithi067.front.sepia.ceph.com:48536 (ESTABLISHED)

2022-04-03T14:13:12.977 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.841 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39705 tp_osd_tp           root   37u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.841 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39705 tp_osd_tp           root   38u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:35.841 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39705 tp_osd_tp           root   39u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.978 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.875 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39707 tp_osd_tp           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:35.876 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39707 tp_osd_tp           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2
, 2022-04-03T01:40:35.876 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39707 tp_osd_tp           root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11

2022-04-03T14:13:12.979 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.910 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39707 tp_osd_tp           root   85u     IPv4             197122         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi067.front.sepia.ceph.com:35856 (ESTABLISHED)
, 2022-04-03T01:40:35.910 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39707 tp_osd_tp           root   86u     IPv4             197125         0t0           TCP smithi061.front.sepia.ceph.com:6814->smithi061.front.sepia.ceph.com:49370 (ESTABLISHED)
, 2022-04-03T01:40:35.911 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39707 tp_osd_tp           root   87u     IPv4             197128         0t0           TCP smithi061.front.sepia.ceph.com:6812->smithi061.front.sepia.ceph.com:44936 (ESTABLISHED)

2022-04-03T14:13:12.979 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39709 tp_osd_tp           root   25u     IPv4             194215         0t0           TCP *:6814 (LISTEN)
, 2022-04-03T01:40:35.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39709 tp_osd_tp           root   26u     IPv4             194231         0t0           TCP *:6815 (LISTEN)
, 2022-04-03T01:40:35.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39709 tp_osd_tp           root   27r     FIFO               0,12         0t0        195971 pipe

2022-04-03T14:13:12.980 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:35.986 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39711 tp_osd_tp           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:35.986 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39711 tp_osd_tp           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:35.987 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39711 tp_osd_tp           root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1

2022-04-03T14:13:12.981 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.020 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39711 tp_osd_tp           root   73u     IPv4             201180         0t0           TCP smithi061.front.sepia.ceph.com:39966->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)
, 2022-04-03T01:40:36.021 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39711 tp_osd_tp           root   74u     IPv4             197105         0t0           TCP smithi061.front.sepia.ceph.com:33366->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:36.021 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39711 tp_osd_tp           root   75u     IPv4             197106         0t0           TCP smithi061.front.sepia.ceph.com:51300->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)

2022-04-03T14:13:12.981 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.054 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39713 tp_osd_tp           root   20u     IPv4             194150         0t0           TCP *:6809 (LISTEN)
, 2022-04-03T01:40:36.054 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39713 tp_osd_tp           root   21u     IPv4             194161         0t0           TCP *:6810 (LISTEN)
, 2022-04-03T01:40:36.055 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39713 tp_osd_tp           root   22u     IPv4             194173         0t0           TCP *:6811 (LISTEN)

2022-04-03T14:13:12.982 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.088 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39715 tp_osd_tp           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:36.088 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39715 tp_osd_tp           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0
, 2022-04-03T01:40:36.088 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39715 tp_osd_tp           root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0

2022-04-03T14:13:12.983 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.122 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39715 tp_osd_tp           root   68u     IPv4             199516         0t0           TCP smithi061.front.sepia.ceph.com:40144->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:36.122 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39715 tp_osd_tp           root   69u     IPv4             201178         0t0           TCP smithi061.front.sepia.ceph.com:32954->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:36.122 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39715 tp_osd_tp           root   70u     IPv4             199517         0t0           TCP smithi061.front.sepia.ceph.com:60140->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)

2022-04-03T14:13:12.983 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.156 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39717 tp_osd_tp           root   14w      REG                8,1  3960835055        790594 /var/log/ceph/ceph-osd.2.log
, 2022-04-03T01:40:36.156 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39717 tp_osd_tp           root   16r     FIFO               0,12         0t0        195963 pipe
, 2022-04-03T01:40:36.156 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39717 tp_osd_tp           root   17w     FIFO               0,12         0t0        195963 pipe

2022-04-03T14:13:12.984 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39719 tp_osd_tp           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:36.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39719 tp_osd_tp           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:36.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39719 tp_osd_tp           root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0

2022-04-03T14:13:12.984 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.224 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39719 tp_osd_tp           root   63u     IPv4             197101         0t0           TCP smithi061.front.sepia.ceph.com:52980->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:36.224 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39719 tp_osd_tp           root   64u     IPv4             197102         0t0           TCP smithi061.front.sepia.ceph.com:36810->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:36.225 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39719 tp_osd_tp           root   65u     IPv4             197103         0t0           TCP smithi061.front.sepia.ceph.com:44910->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)

2022-04-03T14:13:12.985 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.258 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39721 tp_osd_tp           root    9r     FIFO               0,12         0t0        194056 pipe
, 2022-04-03T01:40:36.258 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39721 tp_osd_tp           root   10w     FIFO               0,12         0t0        194056 pipe
, 2022-04-03T01:40:36.258 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39721 tp_osd_tp           root   11u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:12.986 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.291 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39723 tp_osd_tp           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:36.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39723 tp_osd_tp           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:36.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39723 tp_osd_tp           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0

2022-04-03T14:13:12.986 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39723 tp_osd_tp           root   58u     IPv4             271593         0t0           TCP smithi061.front.sepia.ceph.com:6810->smithi067.front.sepia.ceph.com:49784 (ESTABLISHED)
, 2022-04-03T01:40:36.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39723 tp_osd_tp           root   59w     FIFO               0,12         0t0        201956 pipe
, 2022-04-03T01:40:36.326 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39723 tp_osd_tp           root   60u     IPv4             199512         0t0           TCP smithi061.front.sepia.ceph.com:53398->smithi061.front.sepia.ceph.com:6828 (ESTABLISHED)

2022-04-03T14:13:12.987 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.359 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39725 tp_osd_tp           root    4w     FIFO               0,12         0t0        194054 pipe
, 2022-04-03T01:40:36.360 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39725 tp_osd_tp           root    5u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:36.360 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39725 tp_osd_tp           root    6r     FIFO               0,12         0t0        194055 pipe

2022-04-03T14:13:12.988 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.394 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39727 tp_osd_tp           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0
, 2022-04-03T01:40:36.394 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39727 tp_osd_tp           root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0
, 2022-04-03T01:40:36.394 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39727 tp_osd_tp           root  mem       REG                8,1       70768        789090 /usr/lib64/rados-classes/libcls_hello.so.1.0.0

2022-04-03T14:13:12.988 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.428 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39727 tp_osd_tp           root   53u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:36.428 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39727 tp_osd_tp           root   54u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block
, 2022-04-03T01:40:36.428 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39727 tp_osd_tp           root   55u      REG              253,2 96636764160            70 /var/lib/ceph/osd/ceph-2/block

2022-04-03T14:13:12.989 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.468 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39728 osd_srv_h           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:36.468 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39728 osd_srv_h           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:36.469 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37051 39728 osd_srv_h           root  DEL       REG               0,17                    200078 /[aio]

2022-04-03T14:13:12.990 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.502 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052                           root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:36.503 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052                           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0
, 2022-04-03T01:40:36.503 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052                           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0

2022-04-03T14:13:12.990 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.538 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052                           root   45u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:36.538 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052                           root   46u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:36.538 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052                           root   47u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:12.991 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.572 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37080 log                 root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:36.573 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37080 log                 root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:36.573 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37080 log                 root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3

2022-04-03T14:13:12.992 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.607 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37080 log                 root   93u     IPv4             197148         0t0           TCP smithi061.front.sepia.ceph.com:49920->smithi067.front.sepia.ceph.com:6818 (ESTABLISHED)
, 2022-04-03T01:40:36.607 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37080 log                 root   95u     IPv4             198554         0t0           TCP smithi061.front.sepia.ceph.com:33034->smithi061.front.sepia.ceph.com:6820 (ESTABLISHED)
, 2022-04-03T01:40:36.607 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37080 log                 root   98u     IPv4             198555         0t0           TCP smithi061.front.sepia.ceph.com:47452->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)

2022-04-03T14:13:12.992 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.641 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37088 msgr-work           root   38u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:36.641 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37088 msgr-work           root   39u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:36.641 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37088 msgr-work           root   40u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:12.993 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.675 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37089 msgr-work           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:36.676 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37089 msgr-work           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2
, 2022-04-03T01:40:36.676 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37089 msgr-work           root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11

2022-04-03T14:13:12.994 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.710 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37089 msgr-work           root   86u     IPv4             196258         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49470 (ESTABLISHED)
, 2022-04-03T01:40:36.711 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37089 msgr-work           root   87u     IPv4             196259         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49484 (ESTABLISHED)
, 2022-04-03T01:40:36.711 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37089 msgr-work           root   88u     IPv4             196260         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51300 (ESTABLISHED)

2022-04-03T14:13:12.994 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.745 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37090 msgr-work           root   31r      DIR              253,0         278            64 /var/lib/ceph/osd/ceph-0
, 2022-04-03T01:40:36.745 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37090 msgr-work           root   32uW     REG              253,0          37            69 /var/lib/ceph/osd/ceph-0/fsid
, 2022-04-03T01:40:36.746 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37090 msgr-work           root   33u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:12.995 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.780 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37100 service             root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:36.780 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37100 service             root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1
, 2022-04-03T01:40:36.780 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37100 service             root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0

2022-04-03T14:13:12.996 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.814 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37100 service             root   79u     IPv4             193315         0t0           TCP smithi061.front.sepia.ceph.com:40148->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:36.814 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37100 service             root   80u     IPv4             198391         0t0           TCP smithi061.front.sepia.ceph.com:60144->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)
, 2022-04-03T01:40:36.814 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37100 service             root   81u     IPv4             196253         0t0           TCP smithi061.front.sepia.ceph.com:44936->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)

2022-04-03T14:13:12.996 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.847 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37101 admin_soc           root   24u     IPv4             199119         0t0           TCP *:6827 (LISTEN)
, 2022-04-03T01:40:36.847 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37101 admin_soc           root   25u     IPv4             199149         0t0           TCP *:6829 (LISTEN)
, 2022-04-03T01:40:36.848 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37101 admin_soc           root   26u     IPv4             199180         0t0           TCP *:6830 (LISTEN)

2022-04-03T14:13:12.997 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.883 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37148 signal_ha           root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:36.883 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37148 signal_ha           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0
, 2022-04-03T01:40:36.883 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37148 signal_ha           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1

2022-04-03T14:13:12.998 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.917 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37148 signal_ha           root   72u     IPv4             198388         0t0           TCP smithi061.front.sepia.ceph.com:34576->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:36.917 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37148 signal_ha           root   73u     IPv4             198389         0t0           TCP smithi061.front.sepia.ceph.com:49370->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:36.918 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37148 signal_ha           root   74u     IPv4             196251         0t0           TCP smithi061.front.sepia.ceph.com:52984->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)

2022-04-03T14:13:12.998 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.958 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37149 io_contex           root   10w     FIFO               0,12         0t0        189438 pipe
, 2022-04-03T01:40:36.958 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37149 io_contex           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:36.959 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37149 io_contex           root   12r     FIFO               0,12         0t0        189439 pipe

2022-04-03T14:13:12.999 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:36.992 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37151 io_contex           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:36.992 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37151 io_contex           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:36.993 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37151 io_contex           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0

2022-04-03T14:13:13.000 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.027 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37151 io_contex           root   59w     FIFO               0,12         0t0        201172 pipe
, 2022-04-03T01:40:37.028 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37151 io_contex           root   60u     IPv4             198380         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49456 (ESTABLISHED)
, 2022-04-03T01:40:37.028 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37151 io_contex           root   61r     FIFO               0,12         0t0        201173 pipe

2022-04-03T14:13:13.000 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.061 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37168 OpHistory           root    3r     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:37.062 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37168 OpHistory           root    4w     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:37.062 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37168 OpHistory           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.001 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.096 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37169 ceph_time           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:37.096 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37169 ceph_time           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:37.097 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37169 ceph_time           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0

2022-04-03T14:13:13.002 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.130 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37169 ceph_time           root   51u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.130 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37169 ceph_time           root   52u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.131 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37169 ceph_time           root   53u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.002 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.164 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37203 ceph-osd            root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:37.164 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37203 ceph-osd            root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:37.164 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37203 ceph-osd            root  DEL       REG               0,17                    197981 /[aio]

2022-04-03T14:13:13.003 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.198 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37239 ceph-osd            root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:37.198 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37239 ceph-osd            root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:37.198 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37239 ceph-osd            root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0

2022-04-03T14:13:13.004 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.232 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37239 ceph-osd            root   44u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.232 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37239 ceph-osd            root   45u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.232 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37239 ceph-osd            root   46u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.004 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.266 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37253 ceph-osd            root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:37.266 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37253 ceph-osd            root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:37.267 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37253 ceph-osd            root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11

2022-04-03T14:13:13.005 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.300 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37253 ceph-osd            root   92u     IPv4             198396         0t0           TCP smithi061.front.sepia.ceph.com:6821->smithi061.front.sepia.ceph.com:45840 (ESTABLISHED)
, 2022-04-03T01:40:37.300 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37253 ceph-osd            root   93u     IPv4             197148         0t0           TCP smithi061.front.sepia.ceph.com:49920->smithi067.front.sepia.ceph.com:6818 (ESTABLISHED)
, 2022-04-03T01:40:37.301 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37253 ceph-osd            root   95u     IPv4             198554         0t0           TCP smithi061.front.sepia.ceph.com:33034->smithi061.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.006 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.334 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37268 ceph-osd            root   37u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.334 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37268 ceph-osd            root   38u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.334 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37268 ceph-osd            root   39u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.007 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.368 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37282 ceph-osd            root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:37.368 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37282 ceph-osd            root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:37.368 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37282 ceph-osd            root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:13.008 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.402 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37282 ceph-osd            root   85u     IPv4             196257         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51298 (ESTABLISHED)
, 2022-04-03T01:40:37.402 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37282 ceph-osd            root   86u     IPv4             196258         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49470 (ESTABLISHED)
, 2022-04-03T01:40:37.403 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37282 ceph-osd            root   87u     IPv4             196259         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49484 (ESTABLISHED)

2022-04-03T14:13:13.008 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.442 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37296 ceph-osd            root   26u     IPv4             199180         0t0           TCP *:6830 (LISTEN)
, 2022-04-03T01:40:37.443 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37296 ceph-osd            root   27r     FIFO               0,12         0t0        199777 pipe
, 2022-04-03T01:40:37.443 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37296 ceph-osd            root   28w     FIFO               0,12         0t0        199777 pipe

2022-04-03T14:13:13.009 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.477 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37311 ceph-osd            root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:37.478 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37311 ceph-osd            root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:37.478 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37311 ceph-osd            root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1

2022-04-03T14:13:13.010 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.512 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37311 ceph-osd            root   74u     IPv4             196251         0t0           TCP smithi061.front.sepia.ceph.com:52984->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:37.513 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37311 ceph-osd            root   75u     IPv4             193313         0t0           TCP smithi061.front.sepia.ceph.com:34848->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)
, 2022-04-03T01:40:37.513 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37311 ceph-osd            root   76u     IPv4             198390         0t0           TCP smithi061.front.sepia.ceph.com:36814->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)

2022-04-03T14:13:13.010 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.547 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37524 ceph-osd            root   19u     IPv4             194249         0t0           TCP *:pentbox-sim (LISTEN)
, 2022-04-03T01:40:37.547 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37524 ceph-osd            root   20u     IPv4             199769         0t0           TCP *:6819 (LISTEN)
, 2022-04-03T01:40:37.547 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37524 ceph-osd            root   21u     IPv4             199041         0t0           TCP *:6821 (LISTEN)

2022-04-03T14:13:13.011 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.580 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37525 safe_time           root  mem       REG                8,1       57160        789048 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0
, 2022-04-03T01:40:37.581 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37525 safe_time           root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:37.581 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37525 safe_time           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so

2022-04-03T14:13:13.012 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.614 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37525 safe_time           root   67u     IPv4             196245         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59150 (ESTABLISHED)
, 2022-04-03T01:40:37.614 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37525 safe_time           root   68u     IPv4             196246         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59172 (ESTABLISHED)
, 2022-04-03T01:40:37.615 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37525 safe_time           root   69u     IPv4             196247         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi061.front.sepia.ceph.com:39966 (ESTABLISHED)

2022-04-03T14:13:13.012 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.647 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37526 safe_time           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:37.647 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37526 safe_time           root   12r     FIFO               0,12         0t0        189439 pipe
, 2022-04-03T01:40:37.648 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37526 safe_time           root   13w     FIFO               0,12         0t0        189439 pipe

2022-04-03T14:13:13.013 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.681 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37527 safe_time           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:37.682 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37527 safe_time           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:37.682 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37527 safe_time           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0

2022-04-03T14:13:13.014 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37527 safe_time           root   60u     IPv4             198380         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49456 (ESTABLISHED)
, 2022-04-03T01:40:37.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37527 safe_time           root   61r     FIFO               0,12         0t0        201173 pipe
, 2022-04-03T01:40:37.715 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37527 safe_time           root   62w     FIFO               0,12         0t0        201173 pipe

2022-04-03T14:13:13.014 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.749 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37528 safe_time           root    4w     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:37.749 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37528 safe_time           root    5u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:37.750 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37528 safe_time           root    6r     FIFO               0,12         0t0        189437 pipe

2022-04-03T14:13:13.015 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.783 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37529 fn_anonym           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:37.783 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37529 fn_anonym           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0
, 2022-04-03T01:40:37.784 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37529 fn_anonym           root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0

2022-04-03T14:13:13.016 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.817 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37529 fn_anonym           root   52u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.817 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37529 fn_anonym           root   53u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.817 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37529 fn_anonym           root   54u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.016 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.857 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37591 rocksdb:l           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:37.857 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37591 rocksdb:l           root  DEL       REG               0,17                    197981 /[aio]
, 2022-04-03T01:40:37.858 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37591 rocksdb:l           root  DEL       REG               0,17                    197966 /[aio]

2022-04-03T14:13:13.017 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.900 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37594 rocksdb:l           root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:37.901 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37594 rocksdb:l           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0
, 2022-04-03T01:40:37.901 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37594 rocksdb:l           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0

2022-04-03T14:13:13.018 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.951 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37594 rocksdb:l           root   39u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.952 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37594 rocksdb:l           root   40u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:37.952 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37594 rocksdb:l           root   41u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.018 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:37.986 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37596 rocksdb:h           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2
, 2022-04-03T01:40:37.986 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37596 rocksdb:h           root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11
, 2022-04-03T01:40:37.986 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37596 rocksdb:h           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so

2022-04-03T14:13:13.019 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.019 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37596 rocksdb:h           root   87u     IPv4             196259         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49484 (ESTABLISHED)
, 2022-04-03T01:40:38.020 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37596 rocksdb:h           root   88u     IPv4             196260         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51300 (ESTABLISHED)
, 2022-04-03T01:40:38.020 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 37596 rocksdb:h           root   89u     IPv4             196261         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51302 (ESTABLISHED)

2022-04-03T14:13:13.020 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.053 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39155 bstore_ai           root   32uW     REG              253,0          37            69 /var/lib/ceph/osd/ceph-0/fsid
, 2022-04-03T01:40:38.054 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39155 bstore_ai           root   33u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.054 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39155 bstore_ai           root   34u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.020 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.087 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39156 bstore_di           root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1
, 2022-04-03T01:40:38.088 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39156 bstore_di           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:38.088 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39156 bstore_di           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so

2022-04-03T14:13:13.021 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.121 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39156 bstore_di           root   80u     IPv4             198391         0t0           TCP smithi061.front.sepia.ceph.com:60144->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)
, 2022-04-03T01:40:38.122 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39156 bstore_di           root   81u     IPv4             196253         0t0           TCP smithi061.front.sepia.ceph.com:44936->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:38.122 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39156 bstore_di           root   82u     IPv4             196254         0t0           TCP smithi061.front.sepia.ceph.com:44914->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)

2022-04-03T14:13:13.022 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.155 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39791 bstore_ai           root   25u     IPv4             199149         0t0           TCP *:6829 (LISTEN)
, 2022-04-03T01:40:38.156 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39791 bstore_ai           root   26u     IPv4             199180         0t0           TCP *:6830 (LISTEN)
, 2022-04-03T01:40:38.156 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39791 bstore_ai           root   27r     FIFO               0,12         0t0        199777 pipe

2022-04-03T14:13:13.022 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39792 bstore_di           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0
, 2022-04-03T01:40:38.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39792 bstore_di           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:38.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39792 bstore_di           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so

2022-04-03T14:13:13.023 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.224 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39792 bstore_di           root   73u     IPv4             198389         0t0           TCP smithi061.front.sepia.ceph.com:49370->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:38.224 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39792 bstore_di           root   74u     IPv4             196251         0t0           TCP smithi061.front.sepia.ceph.com:52984->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:38.224 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 39792 bstore_di           root   75u     IPv4             193313         0t0           TCP smithi061.front.sepia.ceph.com:34848->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.024 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.257 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40155 ceph-osd            root   18u     unix 0xffff88817a337800         0t0        199683 /var/run/ceph/ceph-osd.0.asok type=STREAM
, 2022-04-03T01:40:38.258 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40155 ceph-osd            root   19u     IPv4             194249         0t0           TCP *:pentbox-sim (LISTEN)
, 2022-04-03T01:40:38.258 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40155 ceph-osd            root   20u     IPv4             199769         0t0           TCP *:6819 (LISTEN)

2022-04-03T14:13:13.024 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40157 cfin                root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0
, 2022-04-03T01:40:38.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40157 cfin                root  mem       REG                8,1       57160        789048 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0
, 2022-04-03T01:40:38.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40157 cfin                root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so

2022-04-03T14:13:13.025 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40157 cfin                root   66u     IPv4             196244         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi061.front.sepia.ceph.com:39964 (ESTABLISHED)
, 2022-04-03T01:40:38.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40157 cfin                root   67u     IPv4             196245         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59150 (ESTABLISHED)
, 2022-04-03T01:40:38.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40157 cfin                root   68u     IPv4             196246         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59172 (ESTABLISHED)

2022-04-03T14:13:13.026 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.358 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40158 bstore_kv           root   10w     FIFO               0,12         0t0        189438 pipe
, 2022-04-03T01:40:38.359 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40158 bstore_kv           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:38.359 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40158 bstore_kv           root   12r     FIFO               0,12         0t0        189439 pipe

2022-04-03T14:13:13.026 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.392 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40159 bstore_kv           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:38.392 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40159 bstore_kv           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:38.393 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40159 bstore_kv           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0

2022-04-03T14:13:13.027 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.425 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40159 bstore_kv           root   59w     FIFO               0,12         0t0        201172 pipe
, 2022-04-03T01:40:38.426 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40159 bstore_kv           root   60u     IPv4             198380         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49456 (ESTABLISHED)
, 2022-04-03T01:40:38.426 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40159 bstore_kv           root   61r     FIFO               0,12         0t0        201173 pipe

2022-04-03T14:13:13.028 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.464 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40160 bstore_me           root  DEL       REG               0,17                    197966 /[aio]
, 2022-04-03T01:40:38.464 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40160 bstore_me           root    0r      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:38.465 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40160 bstore_me           root    1w     FIFO               0,12         0t0        192761 pipe

2022-04-03T14:13:13.028 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.499 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40165 ms_dispat           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0
, 2022-04-03T01:40:38.499 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40165 ms_dispat           root  mem       REG                8,1     1338272        789117 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0
, 2022-04-03T01:40:38.500 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40165 ms_dispat           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0

2022-04-03T14:13:13.029 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.533 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40165 ms_dispat           root   47u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.533 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40165 ms_dispat           root   48u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.534 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40165 ms_dispat           root   49u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.030 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.567 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40167 ms_local            root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3
, 2022-04-03T01:40:38.568 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40167 ms_local            root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:38.568 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40167 ms_local            root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1

2022-04-03T14:13:13.030 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.601 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40167 ms_local            root   98u     IPv4             198555         0t0           TCP smithi061.front.sepia.ceph.com:47452->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)
, 2022-04-03T01:40:38.602 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40167 ms_local            root   99u     IPv4             199606         0t0           TCP smithi061.front.sepia.ceph.com:6821->smithi067.front.sepia.ceph.com:54866 (ESTABLISHED)
, 2022-04-03T01:40:38.602 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40167 ms_local            root  104u     IPv4             241107         0t0           TCP smithi061.front.sepia.ceph.com:47518->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)

2022-04-03T14:13:13.031 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.635 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40170 safe_time           root   40u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.636 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40170 safe_time           root   41u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.636 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40170 safe_time           root   42u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.032 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.669 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40172 safe_time           root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11
, 2022-04-03T01:40:38.669 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40172 safe_time           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so
, 2022-04-03T01:40:38.670 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40172 safe_time           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k

2022-04-03T14:13:13.032 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.703 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40172 safe_time           root   88u     IPv4             196260         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51300 (ESTABLISHED)
, 2022-04-03T01:40:38.703 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40172 safe_time           root   89u     IPv4             196261         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51302 (ESTABLISHED)
, 2022-04-03T01:40:38.703 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40172 safe_time           root   90u     IPv4             197137         0t0           TCP smithi061.front.sepia.ceph.com:59024->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)

2022-04-03T14:13:13.033 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40174 ms_dispat           root   33u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40174 ms_dispat           root   34u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:38.737 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40174 ms_dispat           root   35u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.034 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.770 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40177 ms_local            root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:38.770 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40177 ms_local            root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:38.770 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40177 ms_local            root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1

2022-04-03T14:13:13.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.804 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40177 ms_local            root   81u     IPv4             196253         0t0           TCP smithi061.front.sepia.ceph.com:44936->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:38.804 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40177 ms_local            root   82u     IPv4             196254         0t0           TCP smithi061.front.sepia.ceph.com:44914->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:38.805 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40177 ms_local            root   83u     IPv4             196255         0t0           TCP smithi061.front.sepia.ceph.com:36286->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)

2022-04-03T14:13:13.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.839 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40180 ms_dispat           root   20u     IPv4             199769         0t0           TCP *:6819 (LISTEN)
, 2022-04-03T01:40:38.839 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40180 ms_dispat           root   21u     IPv4             199041         0t0           TCP *:6821 (LISTEN)
, 2022-04-03T01:40:38.840 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40180 ms_dispat           root   22u     IPv4             199065         0t0           TCP *:6823 (LISTEN)

2022-04-03T14:13:13.036 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.874 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40182 ms_local            root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:38.874 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40182 ms_local            root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:38.874 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40182 ms_local            root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:13.037 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.914 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40182 ms_local            root   67u     IPv4             196245         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59150 (ESTABLISHED)
, 2022-04-03T01:40:38.914 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40182 ms_local            root   68u     IPv4             196246         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59172 (ESTABLISHED)
, 2022-04-03T01:40:38.915 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40182 ms_local            root   69u     IPv4             196247         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi061.front.sepia.ceph.com:39966 (ESTABLISHED)

2022-04-03T14:13:13.037 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.947 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40185 ms_dispat           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:38.947 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40185 ms_dispat           root   12r     FIFO               0,12         0t0        189439 pipe
, 2022-04-03T01:40:38.948 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40185 ms_dispat           root   13w     FIFO               0,12         0t0        189439 pipe

2022-04-03T14:13:13.038 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:38.981 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40187 ms_local            root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:38.981 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40187 ms_local            root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:38.981 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40187 ms_local            root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0

2022-04-03T14:13:13.039 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.015 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40187 ms_local            root   60u     IPv4             198380         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49456 (ESTABLISHED)
, 2022-04-03T01:40:39.015 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40187 ms_local            root   61r     FIFO               0,12         0t0        201173 pipe
, 2022-04-03T01:40:39.015 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40187 ms_local            root   62w     FIFO               0,12         0t0        201173 pipe

2022-04-03T14:13:13.039 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.050 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40190 ms_dispat           root    3r     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:39.051 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40190 ms_dispat           root    4w     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:39.051 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40190 ms_dispat           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.040 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.084 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40193 ms_local            root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:39.084 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40193 ms_local            root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:39.084 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40193 ms_local            root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0

2022-04-03T14:13:13.041 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.117 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40193 ms_local            root   51u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.117 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40193 ms_local            root   52u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.118 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40193 ms_local            root   53u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.041 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.151 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40196 ms_dispat           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:39.151 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40196 ms_dispat           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:39.151 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40196 ms_dispat           root  DEL       REG               0,17                    197981 /[aio]

2022-04-03T14:13:13.042 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.185 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40199 ms_local            root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:39.185 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40199 ms_local            root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:39.185 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40199 ms_local            root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0

2022-04-03T14:13:13.043 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.218 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40199 ms_local            root   44u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.218 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40199 ms_local            root   45u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.219 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40199 ms_local            root   46u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.043 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.252 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40202 ms_dispat           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:39.252 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40202 ms_dispat           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:39.252 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40202 ms_dispat           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11

2022-04-03T14:13:13.044 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.285 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40202 ms_dispat           root   92u     IPv4             198396         0t0           TCP smithi061.front.sepia.ceph.com:6821->smithi061.front.sepia.ceph.com:45840 (ESTABLISHED)
, 2022-04-03T01:40:39.285 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40202 ms_dispat           root   93u     IPv4             197148         0t0           TCP smithi061.front.sepia.ceph.com:49920->smithi067.front.sepia.ceph.com:6818 (ESTABLISHED)
, 2022-04-03T01:40:39.286 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40202 ms_dispat           root   95u     IPv4             198554         0t0           TCP smithi061.front.sepia.ceph.com:33034->smithi061.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40208 ms_local            root   37u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40208 ms_local            root   38u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40208 ms_local            root   39u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.351 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40211 fn_anonym           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:39.351 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40211 fn_anonym           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:39.352 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40211 fn_anonym           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:13.046 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.384 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40211 fn_anonym           root   85u     IPv4             196257         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51298 (ESTABLISHED)
, 2022-04-03T01:40:39.384 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40211 fn_anonym           root   86u     IPv4             196258         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49470 (ESTABLISHED)
, 2022-04-03T01:40:39.385 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40211 fn_anonym           root   87u     IPv4             196259         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49484 (ESTABLISHED)

2022-04-03T14:13:13.047 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.419 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40213 finisher            root   29r     FIFO               0,12         0t0        199778 pipe
, 2022-04-03T01:40:39.419 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40213 finisher            root   30w     FIFO               0,12         0t0        199778 pipe
, 2022-04-03T01:40:39.419 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40213 finisher            root   31r      DIR              253,0         278            64 /var/lib/ceph/osd/ceph-0

2022-04-03T14:13:13.047 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.452 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40216 safe_time           root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0
, 2022-04-03T01:40:39.453 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40216 safe_time           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:39.453 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40216 safe_time           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0

2022-04-03T14:13:13.048 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.486 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40216 safe_time           root   77u     IPv4             193314         0t0           TCP smithi061.front.sepia.ceph.com:35638->smithi061.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:39.486 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40216 safe_time           root   78u     IPv4             196252         0t0           TCP smithi061.front.sepia.ceph.com:33370->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:39.487 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40216 safe_time           root   79u     IPv4             193315         0t0           TCP smithi061.front.sepia.ceph.com:40148->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)

2022-04-03T14:13:13.049 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.520 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40218 safe_time           root   22u     IPv4             199065         0t0           TCP *:6823 (LISTEN)
, 2022-04-03T01:40:39.520 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40218 safe_time           root   23u     IPv4             199091         0t0           TCP *:6825 (LISTEN)
, 2022-04-03T01:40:39.520 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40218 safe_time           root   24u     IPv4             199119         0t0           TCP *:6827 (LISTEN)

2022-04-03T14:13:13.049 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.553 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40220 ceph-osd            root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0
, 2022-04-03T01:40:39.553 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40220 ceph-osd            root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:39.554 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40220 ceph-osd            root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0

2022-04-03T14:13:13.050 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.586 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40220 ceph-osd            root   70u     IPv4             196248         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi061.front.sepia.ceph.com:39968 (ESTABLISHED)
, 2022-04-03T01:40:39.587 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40220 ceph-osd            root   71u     IPv4             201176         0t0           TCP smithi061.front.sepia.ceph.com:32958->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:39.587 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40220 ceph-osd            root   72u     IPv4             198388         0t0           TCP smithi061.front.sepia.ceph.com:34576->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)

2022-04-03T14:13:13.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.620 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40222 osd_srv_a           root   14w      REG                8,1  7123652425        790592 /var/log/ceph/ceph-osd.0.log
, 2022-04-03T01:40:39.620 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40222 osd_srv_a           root   16r     FIFO               0,12         0t0        199682 pipe
, 2022-04-03T01:40:39.620 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40222 osd_srv_a           root   17w     FIFO               0,12         0t0        199682 pipe

2022-04-03T14:13:13.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.655 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40224 tp_osd_tp           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:39.655 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40224 tp_osd_tp           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:39.656 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40224 tp_osd_tp           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0

2022-04-03T14:13:13.052 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.688 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40224 tp_osd_tp           root   60u     IPv4             198380         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi067.front.sepia.ceph.com:49456 (ESTABLISHED)
, 2022-04-03T01:40:39.689 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40224 tp_osd_tp           root   61r     FIFO               0,12         0t0        201173 pipe
, 2022-04-03T01:40:39.689 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40224 tp_osd_tp           root   62w     FIFO               0,12         0t0        201173 pipe

2022-04-03T14:13:13.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.723 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40226 tp_osd_tp           root    3r     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:39.723 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40226 tp_osd_tp           root    4w     FIFO               0,12         0t0        189436 pipe
, 2022-04-03T01:40:39.724 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40226 tp_osd_tp           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.757 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40228 tp_osd_tp           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:39.757 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40228 tp_osd_tp           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:39.757 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40228 tp_osd_tp           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0

2022-04-03T14:13:13.054 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.790 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40228 tp_osd_tp           root   51u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.791 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40228 tp_osd_tp           root   52u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.791 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40228 tp_osd_tp           root   53u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.055 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.824 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40229 tp_osd_tp           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:39.824 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40229 tp_osd_tp           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:39.825 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40229 tp_osd_tp           root  DEL       REG               0,17                    197981 /[aio]

2022-04-03T14:13:13.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.864 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40229 tp_osd_tp           root   98u     IPv4             198555         0t0           TCP smithi061.front.sepia.ceph.com:47452->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)
, 2022-04-03T01:40:39.865 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40229 tp_osd_tp           root   99u     IPv4             199606         0t0           TCP smithi061.front.sepia.ceph.com:6821->smithi067.front.sepia.ceph.com:54866 (ESTABLISHED)
, 2022-04-03T01:40:39.865 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40229 tp_osd_tp           root  104u     IPv4             241107         0t0           TCP smithi061.front.sepia.ceph.com:47518->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)

2022-04-03T14:13:13.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.898 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40230 tp_osd_tp           root   40u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.899 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40230 tp_osd_tp           root   41u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.899 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40230 tp_osd_tp           root   42u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.932 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40231 tp_osd_tp           root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11
, 2022-04-03T01:40:39.933 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40231 tp_osd_tp           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so
, 2022-04-03T01:40:39.933 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40231 tp_osd_tp           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k

2022-04-03T14:13:13.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.966 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40231 tp_osd_tp           root   88u     IPv4             196260         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51300 (ESTABLISHED)
, 2022-04-03T01:40:39.966 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40231 tp_osd_tp           root   89u     IPv4             196261         0t0           TCP smithi061.front.sepia.ceph.com:6825->smithi061.front.sepia.ceph.com:51302 (ESTABLISHED)
, 2022-04-03T01:40:39.966 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40231 tp_osd_tp           root   90u     IPv4             197137         0t0           TCP smithi061.front.sepia.ceph.com:59024->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)

2022-04-03T14:13:13.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:39.999 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40232 tp_osd_tp           root   33u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:39.999 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40232 tp_osd_tp           root   34u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:40.000 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40232 tp_osd_tp           root   35u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.033 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40233 tp_osd_tp           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:40.033 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40233 tp_osd_tp           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:40.033 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40233 tp_osd_tp           root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1

2022-04-03T14:13:13.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.066 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40233 tp_osd_tp           root   81u     IPv4             196253         0t0           TCP smithi061.front.sepia.ceph.com:44936->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:40.067 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40233 tp_osd_tp           root   82u     IPv4             196254         0t0           TCP smithi061.front.sepia.ceph.com:44914->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:40.067 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40233 tp_osd_tp           root   83u     IPv4             196255         0t0           TCP smithi061.front.sepia.ceph.com:36286->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)

2022-04-03T14:13:13.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.100 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40234 tp_osd_tp           root   26u     IPv4             199180         0t0           TCP *:6830 (LISTEN)
, 2022-04-03T01:40:40.100 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40234 tp_osd_tp           root   27r     FIFO               0,12         0t0        199777 pipe
, 2022-04-03T01:40:40.101 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40234 tp_osd_tp           root   28w     FIFO               0,12         0t0        199777 pipe

2022-04-03T14:13:13.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.133 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40235 tp_osd_tp           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:40.134 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40235 tp_osd_tp           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:40.134 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40235 tp_osd_tp           root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1

2022-04-03T14:13:13.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.167 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40235 tp_osd_tp           root   74u     IPv4             196251         0t0           TCP smithi061.front.sepia.ceph.com:52984->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:40.168 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40235 tp_osd_tp           root   75u     IPv4             193313         0t0           TCP smithi061.front.sepia.ceph.com:34848->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)
, 2022-04-03T01:40:40.168 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40235 tp_osd_tp           root   76u     IPv4             198390         0t0           TCP smithi061.front.sepia.ceph.com:36814->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)

2022-04-03T14:13:13.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.201 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40236 tp_osd_tp           root   19u     IPv4             194249         0t0           TCP *:pentbox-sim (LISTEN)
, 2022-04-03T01:40:40.201 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40236 tp_osd_tp           root   20u     IPv4             199769         0t0           TCP *:6819 (LISTEN)
, 2022-04-03T01:40:40.201 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40236 tp_osd_tp           root   21u     IPv4             199041         0t0           TCP *:6821 (LISTEN)

2022-04-03T14:13:13.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.236 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40239 tp_osd_tp           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:40.236 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40239 tp_osd_tp           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:40.237 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40239 tp_osd_tp           root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0

2022-04-03T14:13:13.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.270 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40239 tp_osd_tp           root   64u     IPv4             196243         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi067.front.sepia.ceph.com:59146 (ESTABLISHED)
, 2022-04-03T01:40:40.270 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40239 tp_osd_tp           root   65u     IPv4             201175         0t0           TCP smithi061.front.sepia.ceph.com:53402->smithi061.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:40.270 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40239 tp_osd_tp           root   66u     IPv4             196244         0t0           TCP smithi061.front.sepia.ceph.com:6829->smithi061.front.sepia.ceph.com:39964 (ESTABLISHED)

2022-04-03T14:13:13.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.303 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40240 tp_osd_tp           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:40.304 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40240 tp_osd_tp           root    9r     FIFO               0,12         0t0        189438 pipe
, 2022-04-03T01:40:40.304 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40240 tp_osd_tp           root   10w     FIFO               0,12         0t0        189438 pipe

2022-04-03T14:13:13.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.336 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40241 tp_osd_tp           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0
, 2022-04-03T01:40:40.337 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40241 tp_osd_tp           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:40.337 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40241 tp_osd_tp           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0

2022-04-03T14:13:13.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.370 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40241 tp_osd_tp           root   56u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:40.370 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40241 tp_osd_tp           root   58r     FIFO               0,12         0t0        201172 pipe
, 2022-04-03T01:40:40.371 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40241 tp_osd_tp           root   59w     FIFO               0,12         0t0        201172 pipe

2022-04-03T14:13:13.066 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.404 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40242 tp_osd_tp           root    1w     FIFO               0,12         0t0        192761 pipe
, 2022-04-03T01:40:40.404 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40242 tp_osd_tp           root    2w     FIFO               0,12         0t0        192762 pipe
, 2022-04-03T01:40:40.405 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40242 tp_osd_tp           root    3r     FIFO               0,12         0t0        189436 pipe

2022-04-03T14:13:13.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.437 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40243 tp_osd_tp           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0
, 2022-04-03T01:40:40.438 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40243 tp_osd_tp           root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:40.438 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40243 tp_osd_tp           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0

2022-04-03T14:13:13.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.473 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40243 tp_osd_tp           root   47u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:40.473 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40243 tp_osd_tp           root   48u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block
, 2022-04-03T01:40:40.474 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40243 tp_osd_tp           root   49u      REG              253,0 96636764160            70 /var/lib/ceph/osd/ceph-0/block

2022-04-03T14:13:13.068 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.506 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40244 osd_srv_h           root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3
, 2022-04-03T01:40:40.506 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40244 osd_srv_h           root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:40.507 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40244 osd_srv_h           root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1

2022-04-03T14:13:13.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.540 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40244 osd_srv_h           root   98u     IPv4             198555         0t0           TCP smithi061.front.sepia.ceph.com:47452->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)
, 2022-04-03T01:40:40.540 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40244 osd_srv_h           root   99u     IPv4             199606         0t0           TCP smithi061.front.sepia.ceph.com:6821->smithi067.front.sepia.ceph.com:54866 (ESTABLISHED)
, 2022-04-03T01:40:40.540 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37052 40244 osd_srv_h           root  104u     IPv4             241107         0t0           TCP smithi061.front.sepia.ceph.com:47518->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)

2022-04-03T14:13:13.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.572 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053                           root   40u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:40.572 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053                           root   41u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:40.572 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053                           root   42u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.070 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.604 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37056 log                 root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11
, 2022-04-03T01:40:40.604 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37056 log                 root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so
, 2022-04-03T01:40:40.605 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37056 log                 root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k

2022-04-03T14:13:13.071 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.637 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37056 log                 root   87u     IPv4             201184         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35638 (ESTABLISHED)
, 2022-04-03T01:40:40.637 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37056 log                 root   88u     IPv4             201187         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36816 (ESTABLISHED)
, 2022-04-03T01:40:40.637 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37056 log                 root   89u     IPv4             201190         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35642 (ESTABLISHED)

2022-04-03T14:13:13.071 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.669 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37058 msgr-work           root   33u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:40.669 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37058 msgr-work           root   34u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:40.670 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37058 msgr-work           root   35u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.702 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37059 msgr-work           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:40.702 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37059 msgr-work           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:40.702 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37059 msgr-work           root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1

2022-04-03T14:13:13.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.735 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37059 msgr-work           root   80u     IPv4             198387         0t0           TCP smithi061.front.sepia.ceph.com:51302->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:40.735 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37059 msgr-work           root   81u     IPv4             193304         0t0           TCP smithi061.front.sepia.ceph.com:52982->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:40.735 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37059 msgr-work           root   82u     IPv4             193305         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi067.front.sepia.ceph.com:52878 (ESTABLISHED)

2022-04-03T14:13:13.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37060 msgr-work           root   26u     IPv4             199002         0t0           TCP *:6807 (LISTEN)
, 2022-04-03T01:40:40.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37060 msgr-work           root   27r     FIFO               0,12         0t0        192914 pipe
, 2022-04-03T01:40:40.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37060 msgr-work           root   28w     FIFO               0,12         0t0        192914 pipe

2022-04-03T14:13:13.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.800 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37064 service             root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:40.801 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37064 service             root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:40.801 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37064 service             root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1

2022-04-03T14:13:13.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.833 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37064 service             root   73u     IPv4             201967         0t0           TCP smithi061.front.sepia.ceph.com:34846->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)
, 2022-04-03T01:40:40.833 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37064 service             root   74u     IPv4             193301         0t0           TCP smithi061.front.sepia.ceph.com:32956->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:40.834 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37064 service             root   75u     IPv4             201968         0t0           TCP smithi061.front.sepia.ceph.com:53400->smithi061.front.sepia.ceph.com:6828 (ESTABLISHED)

2022-04-03T14:13:13.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.868 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37065 admin_soc           root   16r     FIFO               0,12         0t0        195958 pipe
, 2022-04-03T01:40:40.869 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37065 admin_soc           root   17w     FIFO               0,12         0t0        195958 pipe
, 2022-04-03T01:40:40.869 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37065 admin_soc           root   18u     unix 0xffff888170531680         0t0        195959 /var/run/ceph/ceph-osd.3.asok type=STREAM

2022-04-03T14:13:13.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.900 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37138 signal_ha           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:40.901 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37138 signal_ha           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:40.901 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37138 signal_ha           root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0

2022-04-03T14:13:13.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.933 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37138 signal_ha           root   63u     IPv4             199511         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35610 (ESTABLISHED)
, 2022-04-03T01:40:40.933 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37138 signal_ha           root   64u     IPv4             193288         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35636 (ESTABLISHED)
, 2022-04-03T01:40:40.933 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37138 signal_ha           root   65u     IPv4             193291         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36810 (ESTABLISHED)

2022-04-03T14:13:13.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.965 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37139 io_contex           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:40.965 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37139 io_contex           root    9r     FIFO               0,12         0t0        195954 pipe
, 2022-04-03T01:40:40.966 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37139 io_contex           root   10w     FIFO               0,12         0t0        195954 pipe

2022-04-03T14:13:13.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:40.999 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37140 io_contex           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0
, 2022-04-03T01:40:40.999 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37140 io_contex           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:40.999 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37140 io_contex           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0

2022-04-03T14:13:13.079 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.031 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37140 io_contex           root   56u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.031 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37140 io_contex           root   57u     IPv4             247994         0t0           TCP smithi061.front.sepia.ceph.com:47522->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:41.031 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37140 io_contex           root   58r     FIFO               0,12         0t0        198368 pipe

2022-04-03T14:13:13.079 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.062 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37160 OpHistory           root    1w     FIFO               0,12         0t0        196707 pipe
, 2022-04-03T01:40:41.062 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37160 OpHistory           root    2w     FIFO               0,12         0t0        196708 pipe
, 2022-04-03T01:40:41.063 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37160 OpHistory           root    3r     FIFO               0,12         0t0        195952 pipe

2022-04-03T14:13:13.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.094 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37161 ceph_time           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0
, 2022-04-03T01:40:41.095 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37161 ceph_time           root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:41.095 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37161 ceph_time           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0

2022-04-03T14:13:13.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.128 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37161 ceph_time           root   45u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.128 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37161 ceph_time           root   46u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.129 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37161 ceph_time           root   47u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.160 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37188 ceph-osd            root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:41.160 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37188 ceph-osd            root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:41.160 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37188 ceph-osd            root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3

2022-04-03T14:13:13.082 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37188 ceph-osd            root   92u     IPv4             198399         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40470 (ESTABLISHED)
, 2022-04-03T01:40:41.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37188 ceph-osd            root   94u     IPv4             193339         0t0           TCP smithi061.front.sepia.ceph.com:33032->smithi061.front.sepia.ceph.com:6820 (ESTABLISHED)
, 2022-04-03T01:40:41.194 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37188 ceph-osd            root   96u     IPv4             218941         0t0           TCP smithi061.front.sepia.ceph.com:47458->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)

2022-04-03T14:13:13.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.227 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37233 ceph-osd            root   38u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.228 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37233 ceph-osd            root   39u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.228 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37233 ceph-osd            root   40u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.260 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37247 ceph-osd            root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:41.260 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37247 ceph-osd            root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2
, 2022-04-03T01:40:41.260 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37247 ceph-osd            root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11

2022-04-03T14:13:13.085 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37247 ceph-osd            root   85u     IPv4             193310         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35676 (ESTABLISHED)
, 2022-04-03T01:40:41.292 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37247 ceph-osd            root   86u     IPv4             201181         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36814 (ESTABLISHED)
, 2022-04-03T01:40:41.293 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37247 ceph-osd            root   87u     IPv4             201184         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35638 (ESTABLISHED)

2022-04-03T14:13:13.086 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.324 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37262 ceph-osd            root   31r      DIR              253,3         278            64 /var/lib/ceph/osd/ceph-3
, 2022-04-03T01:40:41.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37262 ceph-osd            root   32uW     REG              253,3          37            69 /var/lib/ceph/osd/ceph-3/fsid
, 2022-04-03T01:40:41.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37262 ceph-osd            root   33u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.363 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37277 ceph-osd            root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:41.363 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37277 ceph-osd            root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:41.363 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37277 ceph-osd            root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1

2022-04-03T14:13:13.088 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.395 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37277 ceph-osd            root   77u     IPv4             193302         0t0           TCP smithi061.front.sepia.ceph.com:36284->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:41.396 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37277 ceph-osd            root   78u     IPv4             198386         0t0           TCP smithi061.front.sepia.ceph.com:33368->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:41.396 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37277 ceph-osd            root   79u     IPv4             193303         0t0           TCP smithi061.front.sepia.ceph.com:39968->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)

2022-04-03T14:13:13.088 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.429 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37293 ceph-osd            root   23u     IPv4             198981         0t0           TCP *:6804 (LISTEN)
, 2022-04-03T01:40:41.429 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37293 ceph-osd            root   24u     IPv4             198987         0t0           TCP *:6805 (LISTEN)
, 2022-04-03T01:40:41.430 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37293 ceph-osd            root   25u     IPv4             198994         0t0           TCP *:6806 (LISTEN)

2022-04-03T14:13:13.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.462 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37308 ceph-osd            root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:41.462 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37308 ceph-osd            root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:41.463 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37308 ceph-osd            root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0

2022-04-03T14:13:13.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.495 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37308 ceph-osd            root   70u     IPv4             193300         0t0           TCP smithi061.front.sepia.ceph.com:44934->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:41.495 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37308 ceph-osd            root   71u     IPv4             201965         0t0           TCP smithi061.front.sepia.ceph.com:40146->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:41.496 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37308 ceph-osd            root   72u     IPv4             201966         0t0           TCP smithi061.front.sepia.ceph.com:60142->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)

2022-04-03T14:13:13.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.527 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37455 ceph-osd            root   16r     FIFO               0,12         0t0        195958 pipe
, 2022-04-03T01:40:41.527 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37455 ceph-osd            root   17w     FIFO               0,12         0t0        195958 pipe
, 2022-04-03T01:40:41.528 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37455 ceph-osd            root   18u     unix 0xffff888170531680         0t0        195959 /var/run/ceph/ceph-osd.3.asok type=STREAM

2022-04-03T14:13:13.091 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.560 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37458 safe_time           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:41.560 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37458 safe_time           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:41.561 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37458 safe_time           root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0

2022-04-03T14:13:13.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.593 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37458 safe_time           root   63u     IPv4             199511         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35610 (ESTABLISHED)
, 2022-04-03T01:40:41.593 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37458 safe_time           root   64u     IPv4             193288         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35636 (ESTABLISHED)
, 2022-04-03T01:40:41.593 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37458 safe_time           root   65u     IPv4             193291         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36810 (ESTABLISHED)

2022-04-03T14:13:13.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.626 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37460 safe_time           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:41.626 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37460 safe_time           root    9r     FIFO               0,12         0t0        195954 pipe
, 2022-04-03T01:40:41.627 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37460 safe_time           root   10w     FIFO               0,12         0t0        195954 pipe

2022-04-03T14:13:13.093 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.659 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37462 safe_time           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0
, 2022-04-03T01:40:41.659 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37462 safe_time           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:41.659 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37462 safe_time           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0

2022-04-03T14:13:13.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.691 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37462 safe_time           root   56u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.691 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37462 safe_time           root   57u     IPv4             247994         0t0           TCP smithi061.front.sepia.ceph.com:47522->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:41.692 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37462 safe_time           root   58r     FIFO               0,12         0t0        198368 pipe

2022-04-03T14:13:13.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.724 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37464 safe_time           root    1w     FIFO               0,12         0t0        196707 pipe
, 2022-04-03T01:40:41.725 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37464 safe_time           root    2w     FIFO               0,12         0t0        196708 pipe
, 2022-04-03T01:40:41.725 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37464 safe_time           root    3r     FIFO               0,12         0t0        195952 pipe

2022-04-03T14:13:13.095 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.758 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37466 fn_anonym           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0
, 2022-04-03T01:40:41.758 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37466 fn_anonym           root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:41.758 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37466 fn_anonym           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0

2022-04-03T14:13:13.096 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.790 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37466 fn_anonym           root   49u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.791 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37466 fn_anonym           root   50u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.791 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37466 fn_anonym           root   51u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.096 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.831 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37521 rocksdb:l           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:41.831 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37521 rocksdb:l           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:41.831 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37521 rocksdb:l           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11

2022-04-03T14:13:13.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.864 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37521 rocksdb:l           root   91u     IPv4             193335         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40472 (ESTABLISHED)
, 2022-04-03T01:40:41.865 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37521 rocksdb:l           root   92u     IPv4             198399         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40470 (ESTABLISHED)
, 2022-04-03T01:40:41.865 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37521 rocksdb:l           root   94u     IPv4             193339         0t0           TCP smithi061.front.sepia.ceph.com:33032->smithi061.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.903 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37522 rocksdb:l           root   37u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.904 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37522 rocksdb:l           root   38u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:41.904 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37522 rocksdb:l           root   39u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.937 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37523 rocksdb:h           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:41.937 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37523 rocksdb:h           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:41.937 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37523 rocksdb:h           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:13.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:41.970 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37523 rocksdb:h           root   84u     IPv4             193309         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35670 (ESTABLISHED)
, 2022-04-03T01:40:41.970 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37523 rocksdb:h           root   85u     IPv4             193310         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35676 (ESTABLISHED)
, 2022-04-03T01:40:41.970 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 37523 rocksdb:h           root   86u     IPv4             201181         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36814 (ESTABLISHED)

2022-04-03T14:13:13.100 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.002 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39046 bstore_ai           root   30w     FIFO               0,12         0t0        192915 pipe
, 2022-04-03T01:40:42.003 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39046 bstore_ai           root   31r      DIR              253,3         278            64 /var/lib/ceph/osd/ceph-3
, 2022-04-03T01:40:42.003 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39046 bstore_ai           root   32uW     REG              253,3          37            69 /var/lib/ceph/osd/ceph-3/fsid

2022-04-03T14:13:13.100 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.035 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39047 bstore_di           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:42.036 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39047 bstore_di           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:42.036 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39047 bstore_di           root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1

2022-04-03T14:13:13.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.068 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39047 bstore_di           root   77u     IPv4             193302         0t0           TCP smithi061.front.sepia.ceph.com:36284->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:42.068 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39047 bstore_di           root   78u     IPv4             198386         0t0           TCP smithi061.front.sepia.ceph.com:33368->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:42.069 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39047 bstore_di           root   79u     IPv4             193303         0t0           TCP smithi061.front.sepia.ceph.com:39968->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)

2022-04-03T14:13:13.102 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.100 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39790 bstore_ai           root   23u     IPv4             198981         0t0           TCP *:6804 (LISTEN)
, 2022-04-03T01:40:42.101 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39790 bstore_ai           root   24u     IPv4             198987         0t0           TCP *:6805 (LISTEN)
, 2022-04-03T01:40:42.101 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39790 bstore_ai           root   25u     IPv4             198994         0t0           TCP *:6806 (LISTEN)

2022-04-03T14:13:13.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.133 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39793 bstore_di           root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:42.134 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39793 bstore_di           root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:42.134 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39793 bstore_di           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0

2022-04-03T14:13:13.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.166 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39793 bstore_di           root   70u     IPv4             193300         0t0           TCP smithi061.front.sepia.ceph.com:44934->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:42.167 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39793 bstore_di           root   71u     IPv4             201965         0t0           TCP smithi061.front.sepia.ceph.com:40146->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:42.167 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 39793 bstore_di           root   72u     IPv4             201966         0t0           TCP smithi061.front.sepia.ceph.com:60142->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)

2022-04-03T14:13:13.104 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.199 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40156 ceph-osd            root   16r     FIFO               0,12         0t0        195958 pipe
, 2022-04-03T01:40:42.200 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40156 ceph-osd            root   17w     FIFO               0,12         0t0        195958 pipe
, 2022-04-03T01:40:42.200 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40156 ceph-osd            root   18u     unix 0xffff888170531680         0t0        195959 /var/run/ceph/ceph-osd.3.asok type=STREAM

2022-04-03T14:13:13.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.233 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40161 cfin                root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:42.233 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40161 cfin                root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:42.234 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40161 cfin                root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0

2022-04-03T14:13:13.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.266 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40161 cfin                root   63u     IPv4             199511         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35610 (ESTABLISHED)
, 2022-04-03T01:40:42.266 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40161 cfin                root   64u     IPv4             193288         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35636 (ESTABLISHED)
, 2022-04-03T01:40:42.267 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40161 cfin                root   65u     IPv4             193291         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36810 (ESTABLISHED)

2022-04-03T14:13:13.106 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.299 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40162 bstore_kv           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:42.300 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40162 bstore_kv           root    9r     FIFO               0,12         0t0        195954 pipe
, 2022-04-03T01:40:42.300 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40162 bstore_kv           root   10w     FIFO               0,12         0t0        195954 pipe

2022-04-03T14:13:13.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.338 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40163 bstore_kv           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:42.339 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40163 bstore_kv           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:42.339 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40163 bstore_kv           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0

2022-04-03T14:13:13.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.371 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40163 bstore_kv           root   51u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:42.371 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40163 bstore_kv           root   52u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:42.371 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40163 bstore_kv           root   53u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.108 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.403 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40164 bstore_me           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:42.404 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40164 bstore_me           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:42.404 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40164 bstore_me           root  DEL       REG               0,17                    193131 /[aio]

2022-04-03T14:13:13.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.436 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40166 ms_dispat           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:42.437 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40166 ms_dispat           root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:42.437 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40166 ms_dispat           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0

2022-04-03T14:13:13.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.470 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40166 ms_dispat           root   44u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:42.471 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40166 ms_dispat           root   45u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:42.471 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40166 ms_dispat           root   46u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.110 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.504 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40168 ms_local            root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:42.504 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40168 ms_local            root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:42.505 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40168 ms_local            root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11

2022-04-03T14:13:13.111 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.537 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40168 ms_local            root   91u     IPv4             193335         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40472 (ESTABLISHED)
, 2022-04-03T01:40:42.537 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40168 ms_local            root   92u     IPv4             198399         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40470 (ESTABLISHED)
, 2022-04-03T01:40:42.538 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40168 ms_local            root   94u     IPv4             193339         0t0           TCP smithi061.front.sepia.ceph.com:33032->smithi061.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.111 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.570 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40169 safe_time           root   37u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:42.570 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40169 safe_time           root   38u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:42.570 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40169 safe_time           root   39u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.607 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40171 safe_time           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:42.607 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40171 safe_time           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:42.608 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40171 safe_time           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:13.113 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.644 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40171 safe_time           root   84u     IPv4             193309         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35670 (ESTABLISHED)
, 2022-04-03T01:40:42.645 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40171 safe_time           root   85u     IPv4             193310         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35676 (ESTABLISHED)
, 2022-04-03T01:40:42.645 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40171 safe_time           root   86u     IPv4             201181         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36814 (ESTABLISHED)

2022-04-03T14:13:13.113 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.684 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40173 ms_dispat           root   30w     FIFO               0,12         0t0        192915 pipe
, 2022-04-03T01:40:42.684 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40173 ms_dispat           root   31r      DIR              253,3         278            64 /var/lib/ceph/osd/ceph-3
, 2022-04-03T01:40:42.685 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40173 ms_dispat           root   32uW     REG              253,3          37            69 /var/lib/ceph/osd/ceph-3/fsid

2022-04-03T14:13:13.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.727 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40175 ms_local            root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:42.728 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40175 ms_local            root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:42.728 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40175 ms_local            root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1

2022-04-03T14:13:13.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.771 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40175 ms_local            root   77u     IPv4             193302         0t0           TCP smithi061.front.sepia.ceph.com:36284->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:42.772 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40175 ms_local            root   78u     IPv4             198386         0t0           TCP smithi061.front.sepia.ceph.com:33368->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:42.772 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40175 ms_local            root   79u     IPv4             193303         0t0           TCP smithi061.front.sepia.ceph.com:39968->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)

2022-04-03T14:13:13.115 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.818 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40176 ms_dispat           root   23u     IPv4             198981         0t0           TCP *:6804 (LISTEN)
, 2022-04-03T01:40:42.819 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40176 ms_dispat           root   24u     IPv4             198987         0t0           TCP *:6805 (LISTEN)
, 2022-04-03T01:40:42.819 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40176 ms_dispat           root   25u     IPv4             198994         0t0           TCP *:6806 (LISTEN)

2022-04-03T14:13:13.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.868 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40178 ms_local            root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:42.868 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40178 ms_local            root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0
, 2022-04-03T01:40:42.868 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40178 ms_local            root  mem       REG                8,1       57160        789048 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0

2022-04-03T14:13:13.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.913 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40178 ms_local            root   64u     IPv4             193288         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35636 (ESTABLISHED)
, 2022-04-03T01:40:42.913 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40178 ms_local            root   65u     IPv4             193291         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36810 (ESTABLISHED)
, 2022-04-03T01:40:42.914 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40178 ms_local            root   66u     IPv4             193294         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi067.front.sepia.ceph.com:52868 (ESTABLISHED)

2022-04-03T14:13:13.117 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40179 ms_dispat           root    9r     FIFO               0,12         0t0        195954 pipe
, 2022-04-03T01:40:42.953 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40179 ms_dispat           root   10w     FIFO               0,12         0t0        195954 pipe
, 2022-04-03T01:40:42.954 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40179 ms_dispat           root   11u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.118 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:42.988 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40181 ms_local            root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:42.988 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40181 ms_local            root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:42.988 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40181 ms_local            root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0

2022-04-03T14:13:13.118 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.022 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40181 ms_local            root   57u     IPv4             247994         0t0           TCP smithi061.front.sepia.ceph.com:47522->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:43.022 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40181 ms_local            root   58r     FIFO               0,12         0t0        198368 pipe
, 2022-04-03T01:40:43.022 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40181 ms_local            root   59w     FIFO               0,12         0t0        198368 pipe

2022-04-03T14:13:13.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.056 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40183 ms_dispat           root    2w     FIFO               0,12         0t0        196708 pipe
, 2022-04-03T01:40:43.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40183 ms_dispat           root    3r     FIFO               0,12         0t0        195952 pipe
, 2022-04-03T01:40:43.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40183 ms_dispat           root    4w     FIFO               0,12         0t0        195952 pipe

2022-04-03T14:13:13.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.090 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40184 ms_local            root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:43.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40184 ms_local            root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:43.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40184 ms_local            root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3

2022-04-03T14:13:13.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.138 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40184 ms_local            root   50u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.138 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40184 ms_local            root   51u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.139 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40184 ms_local            root   52u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.178 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40186 ms_dispat           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7
, 2022-04-03T01:40:43.178 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40186 ms_dispat           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0
, 2022-04-03T01:40:43.178 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40186 ms_dispat           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so

2022-04-03T14:13:13.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.213 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40188 ms_local            root  cwd       DIR                8,1        4096          4714 /home/ubuntu
, 2022-04-03T01:40:43.213 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40188 ms_local            root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:43.213 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40188 ms_local            root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd

2022-04-03T14:13:13.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.252 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40188 ms_local            root   43u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.252 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40188 ms_local            root   44u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.253 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40188 ms_local            root   45u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.289 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40189 ms_dispat           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:43.289 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40189 ms_dispat           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:43.290 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40189 ms_dispat           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3

2022-04-03T14:13:13.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.323 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40189 ms_dispat           root   90u     IPv4             202991         0t0           TCP smithi061.front.sepia.ceph.com:59030->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)
, 2022-04-03T01:40:43.323 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40189 ms_dispat           root   91u     IPv4             193335         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40472 (ESTABLISHED)
, 2022-04-03T01:40:43.324 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40189 ms_dispat           root   92u     IPv4             198399         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40470 (ESTABLISHED)

2022-04-03T14:13:13.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.357 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40191 ms_local            root   36u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.358 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40191 ms_local            root   37u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.358 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40191 ms_local            root   38u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.399 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40192 fn_anonym           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:43.399 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40192 fn_anonym           root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1
, 2022-04-03T01:40:43.400 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40192 fn_anonym           root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0

2022-04-03T14:13:13.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.434 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40192 fn_anonym           root   74u     IPv4             193301         0t0           TCP smithi061.front.sepia.ceph.com:32956->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:43.434 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40192 fn_anonym           root   75u     IPv4             201968         0t0           TCP smithi061.front.sepia.ceph.com:53400->smithi061.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:43.435 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40192 fn_anonym           root   76u     IPv4             198385         0t0           TCP smithi061.front.sepia.ceph.com:34574->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)

2022-04-03T14:13:13.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.469 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40194 finisher            root   20u     IPv4             198969         0t0           TCP *:acnet (LISTEN)
, 2022-04-03T01:40:43.469 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40194 finisher            root   21u     IPv4             198972         0t0           TCP *:6802 (LISTEN)
, 2022-04-03T01:40:43.470 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40194 finisher            root   22u     IPv4             198976         0t0           TCP *:6803 (LISTEN)

2022-04-03T14:13:13.127 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.504 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40195 safe_time           root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:43.504 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40195 safe_time           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:43.504 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40195 safe_time           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:13.127 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.538 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40195 safe_time           root   67u     IPv4             193297         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35666 (ESTABLISHED)
, 2022-04-03T01:40:43.539 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40195 safe_time           root   68u     IPv4             201964         0t0           TCP smithi061.front.sepia.ceph.com:49368->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:43.539 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40195 safe_time           root   69u     IPv4             198384         0t0           TCP smithi061.front.sepia.ceph.com:44912->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)

2022-04-03T14:13:13.128 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.572 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40197 safe_time           root   12r     FIFO               0,12         0t0        195955 pipe
, 2022-04-03T01:40:43.572 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40197 safe_time           root   13w     FIFO               0,12         0t0        195955 pipe
, 2022-04-03T01:40:43.573 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40197 safe_time           root   14w      REG                8,1 10750446138        790595 /var/log/ceph/ceph-osd.3.log

2022-04-03T14:13:13.129 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.605 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40198 ceph-osd            root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:43.606 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40198 ceph-osd            root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0
, 2022-04-03T01:40:43.606 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40198 ceph-osd            root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0

2022-04-03T14:13:13.129 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.639 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40198 ceph-osd            root   60u     IPv4             199508         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi067.front.sepia.ceph.com:52820 (ESTABLISHED)
, 2022-04-03T01:40:43.639 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40198 ceph-osd            root   61r     FIFO               0,12         0t0        198369 pipe
, 2022-04-03T01:40:43.640 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40198 ceph-osd            root   62w     FIFO               0,12         0t0        198369 pipe

2022-04-03T14:13:13.130 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.673 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40200 osd_srv_a           root    5u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:43.673 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40200 osd_srv_a           root    6r     FIFO               0,12         0t0        195953 pipe
, 2022-04-03T01:40:43.673 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40200 osd_srv_a           root    7w     FIFO               0,12         0t0        195953 pipe

2022-04-03T14:13:13.131 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.705 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40201 tp_osd_tp           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0
, 2022-04-03T01:40:43.705 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40201 tp_osd_tp           root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0
, 2022-04-03T01:40:43.705 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40201 tp_osd_tp           root  mem       REG                8,1       70768        789090 /usr/lib64/rados-classes/libcls_hello.so.1.0.0

2022-04-03T14:13:13.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40201 tp_osd_tp           root   53u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40201 tp_osd_tp           root   54u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.737 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40201 tp_osd_tp           root   55u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40203 tp_osd_tp           root  DEL       REG               0,17                    193131 /[aio]
, 2022-04-03T01:40:43.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40203 tp_osd_tp           root  DEL       REG               0,17                    193013 /[aio]
, 2022-04-03T01:40:43.768 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40203 tp_osd_tp           root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:13.133 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.803 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40204 tp_osd_tp           root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:43.803 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40204 tp_osd_tp           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0
, 2022-04-03T01:40:43.803 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40204 tp_osd_tp           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0

2022-04-03T14:13:13.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.837 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40204 tp_osd_tp           root   45u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.837 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40204 tp_osd_tp           root   46u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.838 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40204 tp_osd_tp           root   47u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40205 tp_osd_tp           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:43.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40205 tp_osd_tp           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:43.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40205 tp_osd_tp           root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3

2022-04-03T14:13:13.135 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.917 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40205 tp_osd_tp           root   88u     IPv4             201187         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi061.front.sepia.ceph.com:36816 (ESTABLISHED)
, 2022-04-03T01:40:43.918 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40205 tp_osd_tp           root   89u     IPv4             201190         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35642 (ESTABLISHED)
, 2022-04-03T01:40:43.918 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40205 tp_osd_tp           root   90u     IPv4             202991         0t0           TCP smithi061.front.sepia.ceph.com:59030->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)

2022-04-03T14:13:13.136 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.951 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40206 tp_osd_tp           root   34u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.952 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40206 tp_osd_tp           root   35u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:43.952 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40206 tp_osd_tp           root   36u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.136 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:43.985 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40207 tp_osd_tp           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:43.985 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40207 tp_osd_tp           root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:43.986 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40207 tp_osd_tp           root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so

2022-04-03T14:13:13.137 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.018 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40207 tp_osd_tp           root   81u     IPv4             193304         0t0           TCP smithi061.front.sepia.ceph.com:52982->smithi061.front.sepia.ceph.com:6824 (ESTABLISHED)
, 2022-04-03T01:40:44.019 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40207 tp_osd_tp           root   82u     IPv4             193305         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi067.front.sepia.ceph.com:52878 (ESTABLISHED)
, 2022-04-03T01:40:44.019 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40207 tp_osd_tp           root   83u     IPv4             193306         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi067.front.sepia.ceph.com:52886 (ESTABLISHED)

2022-04-03T14:13:13.138 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.053 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40209 tp_osd_tp           root   27r     FIFO               0,12         0t0        192914 pipe
, 2022-04-03T01:40:44.053 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40209 tp_osd_tp           root   28w     FIFO               0,12         0t0        192914 pipe
, 2022-04-03T01:40:44.053 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40209 tp_osd_tp           root   29r     FIFO               0,12         0t0        192915 pipe

2022-04-03T14:13:13.138 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.086 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40210 tp_osd_tp           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so
, 2022-04-03T01:40:44.086 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40210 tp_osd_tp           root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1
, 2022-04-03T01:40:44.087 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40210 tp_osd_tp           root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0

2022-04-03T14:13:13.139 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.121 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40210 tp_osd_tp           root   74u     IPv4             193301         0t0           TCP smithi061.front.sepia.ceph.com:32956->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:44.121 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40210 tp_osd_tp           root   75u     IPv4             201968         0t0           TCP smithi061.front.sepia.ceph.com:53400->smithi061.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:44.121 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40210 tp_osd_tp           root   76u     IPv4             198385         0t0           TCP smithi061.front.sepia.ceph.com:34574->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)

2022-04-03T14:13:13.140 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.155 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40212 tp_osd_tp           root   20u     IPv4             198969         0t0           TCP *:acnet (LISTEN)
, 2022-04-03T01:40:44.155 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40212 tp_osd_tp           root   21u     IPv4             198972         0t0           TCP *:6802 (LISTEN)
, 2022-04-03T01:40:44.156 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40212 tp_osd_tp           root   22u     IPv4             198976         0t0           TCP *:6803 (LISTEN)

2022-04-03T14:13:13.140 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.190 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40214 tp_osd_tp           root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:44.191 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40214 tp_osd_tp           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:44.191 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40214 tp_osd_tp           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:13.141 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.225 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40214 tp_osd_tp           root   67u     IPv4             193297         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi067.front.sepia.ceph.com:35666 (ESTABLISHED)
, 2022-04-03T01:40:44.225 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40214 tp_osd_tp           root   68u     IPv4             201964         0t0           TCP smithi061.front.sepia.ceph.com:49368->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:44.226 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40214 tp_osd_tp           root   69u     IPv4             198384         0t0           TCP smithi061.front.sepia.ceph.com:44912->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)

2022-04-03T14:13:13.142 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.260 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40215 tp_osd_tp           root   12r     FIFO               0,12         0t0        195955 pipe
, 2022-04-03T01:40:44.260 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40215 tp_osd_tp           root   13w     FIFO               0,12         0t0        195955 pipe
, 2022-04-03T01:40:44.261 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40215 tp_osd_tp           root   14w      REG                8,1 10750446425        790595 /var/log/ceph/ceph-osd.3.log

2022-04-03T14:13:13.142 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.295 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40217 tp_osd_tp           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:44.295 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40217 tp_osd_tp           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0
, 2022-04-03T01:40:44.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40217 tp_osd_tp           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0

2022-04-03T14:13:13.143 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.330 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40217 tp_osd_tp           root   60u     IPv4             199508         0t0           TCP smithi061.front.sepia.ceph.com:6804->smithi067.front.sepia.ceph.com:52820 (ESTABLISHED)
, 2022-04-03T01:40:44.330 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40217 tp_osd_tp           root   61r     FIFO               0,12         0t0        198369 pipe
, 2022-04-03T01:40:44.331 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40217 tp_osd_tp           root   62w     FIFO               0,12         0t0        198369 pipe

2022-04-03T14:13:13.144 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.365 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40219 tp_osd_tp           root    3r     FIFO               0,12         0t0        195952 pipe
, 2022-04-03T01:40:44.366 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40219 tp_osd_tp           root    4w     FIFO               0,12         0t0        195952 pipe
, 2022-04-03T01:40:44.366 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40219 tp_osd_tp           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.144 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.408 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40221 tp_osd_tp           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0
, 2022-04-03T01:40:44.408 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40221 tp_osd_tp           root  mem       REG                8,1      103600        789096 /usr/lib64/rados-classes/libcls_lock.so.1.0.0
, 2022-04-03T01:40:44.409 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40221 tp_osd_tp           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0

2022-04-03T14:13:13.145 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.443 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40221 tp_osd_tp           root   49u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:44.443 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40221 tp_osd_tp           root   50u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:44.444 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40221 tp_osd_tp           root   51u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.146 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.479 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40223 tp_osd_tp           root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1
, 2022-04-03T01:40:44.479 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40223 tp_osd_tp           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7
, 2022-04-03T01:40:44.479 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40223 tp_osd_tp           root  mem       REG                8,1      343608         15743 /usr/lib64/libblkid.so.1.1.0

2022-04-03T14:13:13.146 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.514 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40223 tp_osd_tp           root   99u     IPv4             193341         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi067.front.sepia.ceph.com:55820 (ESTABLISHED)
, 2022-04-03T01:40:44.514 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40225 tp_osd_tp           root  cwd       DIR                8,1        4096          4714 /home/ubuntu
, 2022-04-03T01:40:44.515 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40225 tp_osd_tp           root  rtd       DIR                8,1        4096             2 /

2022-04-03T14:13:13.147 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.550 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40225 tp_osd_tp           root   42u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:44.550 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40225 tp_osd_tp           root   43u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block
, 2022-04-03T01:40:44.550 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40225 tp_osd_tp           root   44u      REG              253,3 96636764160            70 /var/lib/ceph/osd/ceph-3/block

2022-04-03T14:13:13.148 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.585 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40227 osd_srv_h           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k
, 2022-04-03T01:40:44.585 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40227 osd_srv_h           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:44.585 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40227 osd_srv_h           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so

2022-04-03T14:13:13.149 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.621 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40227 osd_srv_h           root   89u     IPv4             201190         0t0           TCP smithi061.front.sepia.ceph.com:6806->smithi061.front.sepia.ceph.com:35642 (ESTABLISHED)
, 2022-04-03T01:40:44.621 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40227 osd_srv_h           root   90u     IPv4             202991         0t0           TCP smithi061.front.sepia.ceph.com:59030->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)
, 2022-04-03T01:40:44.622 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37053 40227 osd_srv_h           root   91u     IPv4             193335         0t0           TCP smithi061.front.sepia.ceph.com:6802->smithi061.front.sepia.ceph.com:40472 (ESTABLISHED)

2022-04-03T14:13:13.149 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.656 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054                           root   35u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:44.656 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054                           root   36u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:44.657 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054                           root   37u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.150 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.692 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37078 log                 root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so
, 2022-04-03T01:40:44.693 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37078 log                 root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:44.693 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37078 log                 root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2

2022-04-03T14:13:13.151 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.726 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37078 log                 root   83u     IPv4             201963         0t0           TCP smithi061.front.sepia.ceph.com:49372->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:44.727 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37078 log                 root   84u     IPv4             196264         0t0           TCP smithi061.front.sepia.ceph.com:35642->smithi061.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:44.727 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37078 log                 root   85u     IPv4             199543         0t0           TCP smithi061.front.sepia.ceph.com:34850->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.151 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.760 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37081 msgr-work           root   30w     FIFO               0,12         0t0        195241 pipe
, 2022-04-03T01:40:44.761 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37081 msgr-work           root   31r      DIR              253,1         278            64 /var/lib/ceph/osd/ceph-1
, 2022-04-03T01:40:44.761 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37081 msgr-work           root   32uW     REG              253,1          37            69 /var/lib/ceph/osd/ceph-1/fsid

2022-04-03T14:13:13.152 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.795 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37083 msgr-work           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:44.796 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37083 msgr-work           root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1
, 2022-04-03T01:40:44.796 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37083 msgr-work           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0

2022-04-03T14:13:13.153 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.829 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37083 msgr-work           root   78u     IPv4             199537         0t0           TCP smithi061.front.sepia.ceph.com:36816->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:44.830 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37083 msgr-work           root   79u     IPv4             199538         0t0           TCP smithi061.front.sepia.ceph.com:40150->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:44.830 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37083 msgr-work           root   80u     IPv4             199539         0t0           TCP smithi061.front.sepia.ceph.com:60146->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)

2022-04-03T14:13:13.153 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.864 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37084 msgr-work           root   25u     IPv4             194348         0t0           TCP *:6828 (LISTEN)
, 2022-04-03T01:40:44.864 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37084 msgr-work           root   26u     IPv4             194380         0t0           TCP *:ambit-lm (LISTEN)
, 2022-04-03T01:40:44.865 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37084 msgr-work           root   27r     FIFO               0,12         0t0        195240 pipe

2022-04-03T14:13:13.154 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.906 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37096 service             root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:44.906 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37096 service             root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:44.907 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37096 service             root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:13.155 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.940 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37096 service             root   67u     IPv4             199522         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52982 (ESTABLISHED)
, 2022-04-03T01:40:44.941 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37096 service             root   68u     IPv4             199523         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57914 (ESTABLISHED)
, 2022-04-03T01:40:44.941 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37096 service             root   69u     IPv4             199524         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57926 (ESTABLISHED)

2022-04-03T14:13:13.155 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:44.975 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37099 admin_soc           root   13w     FIFO               0,12         0t0        192876 pipe
, 2022-04-03T01:40:44.975 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37099 admin_soc           root   14w      REG                8,1  8409852905        790593 /var/log/ceph/ceph-osd.1.log
, 2022-04-03T01:40:44.976 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37099 admin_soc           root   16r     FIFO               0,12         0t0        197774 pipe

2022-04-03T14:13:13.156 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.010 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37150 signal_ha           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0
, 2022-04-03T01:40:45.010 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37150 signal_ha           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:45.011 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37150 signal_ha           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0

2022-04-03T14:13:13.157 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.045 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37150 signal_ha           root   62w     FIFO               0,12         0t0        196241 pipe
, 2022-04-03T01:40:45.045 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37150 signal_ha           root   63u     IPv4             201177         0t0           TCP smithi061.front.sepia.ceph.com:51298->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:45.045 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37150 signal_ha           root   64u     IPv4             199519         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52980 (ESTABLISHED)

2022-04-03T14:13:13.157 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.079 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37152 io_contex           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:45.079 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37152 io_contex           root    9r     FIFO               0,12         0t0        192875 pipe
, 2022-04-03T01:40:45.080 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37152 io_contex           root   10w     FIFO               0,12         0t0        192875 pipe

2022-04-03T14:13:13.158 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.114 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37153 io_contex           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:45.114 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37153 io_contex           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:45.114 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37153 io_contex           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0

2022-04-03T14:13:13.159 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.148 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37153 io_contex           root   57u     IPv4             243184         0t0           TCP smithi061.front.sepia.ceph.com:47524->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:45.148 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37153 io_contex           root   58u     IPv4             193286         0t0           TCP smithi061.front.sepia.ceph.com:39964->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)
, 2022-04-03T01:40:45.149 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37153 io_contex           root   59r     FIFO               0,12         0t0        196240 pipe

2022-04-03T14:13:13.159 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.182 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37172 OpHistory           root    3r     FIFO               0,12         0t0        192873 pipe
, 2022-04-03T01:40:45.182 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37172 OpHistory           root    4w     FIFO               0,12         0t0        192873 pipe
, 2022-04-03T01:40:45.182 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37172 OpHistory           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.160 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.217 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37173 ceph_time           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:45.217 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37173 ceph_time           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0
, 2022-04-03T01:40:45.217 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37173 ceph_time           root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0

2022-04-03T14:13:13.161 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.253 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37173 ceph_time           root   52u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.253 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37173 ceph_time           root   53u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.253 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37173 ceph_time           root   54u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.161 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.288 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37206 ceph-osd            root  DEL       REG               0,17                    200853 /[aio]
, 2022-04-03T01:40:45.289 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37206 ceph-osd            root  DEL       REG               0,17                    200825 /[aio]
, 2022-04-03T01:40:45.289 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37206 ceph-osd            root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:13.162 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.324 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37243 ceph-osd            root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0
, 2022-04-03T01:40:45.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37243 ceph-osd            root  mem       REG                8,1     1338272        789117 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0
, 2022-04-03T01:40:45.325 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37243 ceph-osd            root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0

2022-04-03T14:13:13.163 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.360 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37243 ceph-osd            root   47u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.361 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37243 ceph-osd            root   48u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.361 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37243 ceph-osd            root   49u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.163 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.397 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37256 ceph-osd            root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:45.398 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37256 ceph-osd            root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1
, 2022-04-03T01:40:45.398 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37256 ceph-osd            root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7

2022-04-03T14:13:13.164 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.442 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37256 ceph-osd            root   91u     IPv4             247734         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi067.front.sepia.ceph.com:42528 (ESTABLISHED)
, 2022-04-03T01:40:45.442 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37256 ceph-osd            root   93u     IPv4             199567         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi061.front.sepia.ceph.com:33032 (ESTABLISHED)
, 2022-04-03T01:40:45.442 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37256 ceph-osd            root   95u     IPv4             221863         0t0           TCP smithi061.front.sepia.ceph.com:47454->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)

2022-04-03T14:13:13.165 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.477 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37271 ceph-osd            root   38u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.477 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37271 ceph-osd            root   39u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.477 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37271 ceph-osd            root   40u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.165 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.513 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37285 ceph-osd            root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2
, 2022-04-03T01:40:45.513 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37285 ceph-osd            root  mem       REG                8,1      636712          1904 /usr/lib64/libudev.so.1.6.11
, 2022-04-03T01:40:45.514 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37285 ceph-osd            root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so

2022-04-03T14:13:13.166 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.549 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37285 ceph-osd            root   86u     IPv4             199544         0t0           TCP smithi061.front.sepia.ceph.com:44916->smithi067.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:45.550 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37285 ceph-osd            root   87u     IPv4             196265         0t0           TCP smithi061.front.sepia.ceph.com:34578->smithi067.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:45.550 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37285 ceph-osd            root   88u     IPv4             199545         0t0           TCP smithi061.front.sepia.ceph.com:32960->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)

2022-04-03T14:13:13.167 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.585 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37299 ceph-osd            root   33u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.586 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37299 ceph-osd            root   34u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:45.586 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37299 ceph-osd            root   35u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.167 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.622 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37314 ceph-osd            root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:45.622 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37314 ceph-osd            root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:45.622 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37314 ceph-osd            root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so

2022-04-03T14:13:13.168 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.658 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37314 ceph-osd            root   81u     IPv4             199540         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52984 (ESTABLISHED)
, 2022-04-03T01:40:45.658 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37314 ceph-osd            root   82u     IPv4             193287         0t0           TCP smithi061.front.sepia.ceph.com:44938->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:45.659 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37314 ceph-osd            root   83u     IPv4             201963         0t0           TCP smithi061.front.sepia.ceph.com:49372->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)

2022-04-03T14:13:13.169 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.694 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37825 ceph-osd            root   28w     FIFO               0,12         0t0        195240 pipe
, 2022-04-03T01:40:45.694 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37825 ceph-osd            root   29r     FIFO               0,12         0t0        195241 pipe
, 2022-04-03T01:40:45.695 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37825 ceph-osd            root   30w     FIFO               0,12         0t0        195241 pipe

2022-04-03T14:13:13.170 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.730 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37831 safe_time           root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0
, 2022-04-03T01:40:45.730 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37831 safe_time           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:45.731 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37831 safe_time           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0

2022-04-03T14:13:13.170 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.767 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37831 safe_time           root   76u     IPv4             199533         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi061.front.sepia.ceph.com:53402 (ESTABLISHED)
, 2022-04-03T01:40:45.767 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37831 safe_time           root   77u     IPv4             199536         0t0           TCP smithi061.front.sepia.ceph.com:33372->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:45.767 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37831 safe_time           root   78u     IPv4             199537         0t0           TCP smithi061.front.sepia.ceph.com:36816->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)

2022-04-03T14:13:13.171 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.802 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37835 safe_time           root   23u     IPv4             194315         0t0           TCP *:6824 (LISTEN)
, 2022-04-03T01:40:45.802 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37835 safe_time           root   24u     IPv4             192987         0t0           TCP *:6826 (LISTEN)
, 2022-04-03T01:40:45.803 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37835 safe_time           root   25u     IPv4             194348         0t0           TCP *:6828 (LISTEN)

2022-04-03T14:13:13.172 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.838 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37838 safe_time           root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:45.839 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37838 safe_time           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0
, 2022-04-03T01:40:45.839 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37838 safe_time           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1

2022-04-03T14:13:13.172 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.874 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37838 safe_time           root   71u     IPv4             199528         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55902 (ESTABLISHED)
, 2022-04-03T01:40:45.875 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37838 safe_time           root   72u     IPv4             199529         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55920 (ESTABLISHED)
, 2022-04-03T01:40:45.875 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37838 safe_time           root   73u     IPv4             199530         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi061.front.sepia.ceph.com:53400 (ESTABLISHED)

2022-04-03T14:13:13.173 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.917 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37840 safe_time           root   18u     unix 0xffff88810e40c380         0t0        197775 /var/run/ceph/ceph-osd.1.asok type=STREAM
, 2022-04-03T01:40:45.917 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37840 safe_time           root   19u     IPv4             192933         0t0           TCP *:6816 (LISTEN)
, 2022-04-03T01:40:45.918 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37840 safe_time           root   20u     IPv4             192952         0t0           TCP *:6818 (LISTEN)

2022-04-03T14:13:13.174 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.960 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37845 fn_anonym           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0
, 2022-04-03T01:40:45.960 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37845 fn_anonym           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:45.961 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37845 fn_anonym           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0

2022-04-03T14:13:13.174 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.995 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37845 fn_anonym           root   62w     FIFO               0,12         0t0        196241 pipe
, 2022-04-03T01:40:45.995 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37845 fn_anonym           root   63u     IPv4             201177         0t0           TCP smithi061.front.sepia.ceph.com:51298->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:45.996 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37845 fn_anonym           root   64u     IPv4             199519         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52980 (ESTABLISHED)

2022-04-03T14:13:13.175 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.032 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37907 rocksdb:l           root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:46.032 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37907 rocksdb:l           root    9r     FIFO               0,12         0t0        192875 pipe
, 2022-04-03T01:40:46.032 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37907 rocksdb:l           root   10w     FIFO               0,12         0t0        192875 pipe

2022-04-03T14:13:13.176 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.067 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37908 rocksdb:l           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:46.067 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37908 rocksdb:l           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:46.068 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37908 rocksdb:l           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0

2022-04-03T14:13:13.177 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.101 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37908 rocksdb:l           root   57u     IPv4             243184         0t0           TCP smithi061.front.sepia.ceph.com:47524->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:46.101 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37908 rocksdb:l           root   58u     IPv4             193286         0t0           TCP smithi061.front.sepia.ceph.com:39964->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)
, 2022-04-03T01:40:46.102 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37908 rocksdb:l           root   59r     FIFO               0,12         0t0        196240 pipe

2022-04-03T14:13:13.177 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.137 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37909 rocksdb:h           root    3r     FIFO               0,12         0t0        192873 pipe
, 2022-04-03T01:40:46.138 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37909 rocksdb:h           root    4w     FIFO               0,12         0t0        192873 pipe
, 2022-04-03T01:40:46.138 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 37909 rocksdb:h           root    5u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.178 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.436 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39566 bstore_ai           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:46.436 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39566 bstore_ai           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0
, 2022-04-03T01:40:46.437 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39566 bstore_ai           root  mem       REG                8,1       74600        789099 /usr/lib64/rados-classes/libcls_log.so.1.0.0

2022-04-03T14:13:13.179 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.484 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39566 bstore_ai           root   52u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:46.484 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39566 bstore_ai           root   53u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:46.484 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39566 bstore_ai           root   54u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.179 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.524 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39567 bstore_di           root  DEL       REG               0,17                    200853 /[aio]
, 2022-04-03T01:40:46.524 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39567 bstore_di           root  DEL       REG               0,17                    200825 /[aio]
, 2022-04-03T01:40:46.524 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 39567 bstore_di           root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:13.180 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.560 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40237 bstore_ai           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0
, 2022-04-03T01:40:46.560 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40237 bstore_ai           root  mem       REG                8,1     1338272        789117 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0
, 2022-04-03T01:40:46.560 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40237 bstore_ai           root  mem       REG                8,1      272736        789075 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0

2022-04-03T14:13:13.181 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.597 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40237 bstore_ai           root   47u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:46.598 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40237 bstore_ai           root   48u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:46.598 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40237 bstore_ai           root   49u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.181 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.634 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40238 bstore_di           root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:46.635 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40238 bstore_di           root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1
, 2022-04-03T01:40:46.635 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40238 bstore_di           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7

2022-04-03T14:13:13.182 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.673 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40238 bstore_di           root   98u     IPv4             201219         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi061.front.sepia.ceph.com:33034 (ESTABLISHED)
, 2022-04-03T01:40:46.674 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40425 ceph-osd            root  cwd       DIR                8,1        4096          4714 /home/ubuntu
, 2022-04-03T01:40:46.674 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40425 ceph-osd            root  rtd       DIR                8,1        4096             2 /

2022-04-03T14:13:13.183 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.709 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40425 ceph-osd            root   42u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:46.710 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40425 ceph-osd            root   43u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:46.710 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40425 ceph-osd            root   44u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.183 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.750 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40426 cfin                root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:46.751 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40426 cfin                root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:46.751 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40426 cfin                root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3

2022-04-03T14:13:13.184 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.793 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40426 cfin                root   82u     IPv4             193287         0t0           TCP smithi061.front.sepia.ceph.com:44938->smithi061.front.sepia.ceph.com:6812 (ESTABLISHED)
, 2022-04-03T01:40:46.794 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40426 cfin                root   83u     IPv4             201963         0t0           TCP smithi061.front.sepia.ceph.com:49372->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:46.794 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40426 cfin                root   84u     IPv4             196264         0t0           TCP smithi061.front.sepia.ceph.com:35642->smithi061.front.sepia.ceph.com:6806 (ESTABLISHED)

2022-04-03T14:13:13.185 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.829 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40427 bstore_kv           root   29r     FIFO               0,12         0t0        195241 pipe
, 2022-04-03T01:40:46.829 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40427 bstore_kv           root   30w     FIFO               0,12         0t0        195241 pipe
, 2022-04-03T01:40:46.829 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40427 bstore_kv           root   31r      DIR              253,1         278            64 /var/lib/ceph/osd/ceph-1

2022-04-03T14:13:13.185 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.871 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40428 bstore_kv           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:46.871 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40428 bstore_kv           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:46.871 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40428 bstore_kv           root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1

2022-04-03T14:13:13.186 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.906 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40428 bstore_kv           root   77u     IPv4             199536         0t0           TCP smithi061.front.sepia.ceph.com:33372->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)
, 2022-04-03T01:40:46.907 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40428 bstore_kv           root   78u     IPv4             199537         0t0           TCP smithi061.front.sepia.ceph.com:36816->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:46.907 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40428 bstore_kv           root   79u     IPv4             199538         0t0           TCP smithi061.front.sepia.ceph.com:40150->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)

2022-04-03T14:13:13.187 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.943 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40429 bstore_me           root   24u     IPv4             192987         0t0           TCP *:6826 (LISTEN)
, 2022-04-03T01:40:46.943 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40429 bstore_me           root   25u     IPv4             194348         0t0           TCP *:6828 (LISTEN)
, 2022-04-03T01:40:46.944 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40429 bstore_me           root   26u     IPv4             194380         0t0           TCP *:ambit-lm (LISTEN)

2022-04-03T14:13:13.187 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.980 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40431 ms_dispat           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0
, 2022-04-03T01:40:46.981 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40431 ms_dispat           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1
, 2022-04-03T01:40:46.981 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40431 ms_dispat           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so

2022-04-03T14:13:13.188 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.017 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40431 ms_dispat           root   72u     IPv4             199529         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55920 (ESTABLISHED)
, 2022-04-03T01:40:47.017 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40431 ms_dispat           root   73u     IPv4             199530         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi061.front.sepia.ceph.com:53400 (ESTABLISHED)
, 2022-04-03T01:40:47.018 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40431 ms_dispat           root   74u     IPv4             199531         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55950 (ESTABLISHED)

2022-04-03T14:13:13.189 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40432 ms_local            root   19u     IPv4             192933         0t0           TCP *:6816 (LISTEN)
, 2022-04-03T01:40:47.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40432 ms_local            root   20u     IPv4             192952         0t0           TCP *:6818 (LISTEN)
, 2022-04-03T01:40:47.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40432 ms_local            root   21u     IPv4             194267         0t0           TCP *:6820 (LISTEN)

2022-04-03T14:13:13.189 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.092 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40433 safe_time           root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:47.093 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40433 safe_time           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:47.093 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40433 safe_time           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0

2022-04-03T14:13:13.190 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.149 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40433 safe_time           root   67u     IPv4             199522         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52982 (ESTABLISHED)
, 2022-04-03T01:40:47.149 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40433 safe_time           root   68u     IPv4             199523         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57914 (ESTABLISHED)
, 2022-04-03T01:40:47.150 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40433 safe_time           root   69u     IPv4             199524         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57926 (ESTABLISHED)

2022-04-03T14:13:13.191 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.192 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40434 safe_time           root   13w     FIFO               0,12         0t0        192876 pipe
, 2022-04-03T01:40:47.192 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40434 safe_time           root   14w      REG                8,1  8409853479        790593 /var/log/ceph/ceph-osd.1.log
, 2022-04-03T01:40:47.192 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40434 safe_time           root   16r     FIFO               0,12         0t0        197774 pipe

2022-04-03T14:13:13.191 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.226 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40435 ms_dispat           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0
, 2022-04-03T01:40:47.226 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40435 ms_dispat           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:47.227 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40435 ms_dispat           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0

2022-04-03T14:13:13.192 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.261 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40435 ms_dispat           root   62w     FIFO               0,12         0t0        196241 pipe
, 2022-04-03T01:40:47.261 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40435 ms_dispat           root   63u     IPv4             201177         0t0           TCP smithi061.front.sepia.ceph.com:51298->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:47.262 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40435 ms_dispat           root   64u     IPv4             199519         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52980 (ESTABLISHED)

2022-04-03T14:13:13.193 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40436 ms_local            root    8u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:47.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40436 ms_local            root    9r     FIFO               0,12         0t0        192875 pipe
, 2022-04-03T01:40:47.296 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40436 ms_local            root   10w     FIFO               0,12         0t0        192875 pipe

2022-04-03T14:13:13.193 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.337 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40437 ms_dispat           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0
, 2022-04-03T01:40:47.337 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40437 ms_dispat           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0
, 2022-04-03T01:40:47.337 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40437 ms_dispat           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0

2022-04-03T14:13:13.194 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.371 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40437 ms_dispat           root   56u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.371 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40437 ms_dispat           root   57u     IPv4             243184         0t0           TCP smithi061.front.sepia.ceph.com:47524->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)
, 2022-04-03T01:40:47.372 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40437 ms_dispat           root   58u     IPv4             193286         0t0           TCP smithi061.front.sepia.ceph.com:39964->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)

2022-04-03T14:13:13.195 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.406 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40438 ms_local            root    2w     FIFO               0,12         0t0        192766 pipe
, 2022-04-03T01:40:47.406 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40438 ms_local            root    3r     FIFO               0,12         0t0        192873 pipe
, 2022-04-03T01:40:47.407 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40438 ms_local            root    4w     FIFO               0,12         0t0        192873 pipe

2022-04-03T14:13:13.195 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.440 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40439 ms_dispat           root  mem       REG                8,1       66088        789132 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0
, 2022-04-03T01:40:47.440 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40439 ms_dispat           root  mem       REG                8,1      101232         24150 /usr/lib64/liboath.so.0.1.3
, 2022-04-03T01:40:47.441 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40439 ms_dispat           root  mem       REG                8,1       95488        789108 /usr/lib64/rados-classes/libcls_otp.so.1.0.0

2022-04-03T14:13:13.196 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.475 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40439 ms_dispat           root   51u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.475 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40439 ms_dispat           root   52u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.475 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40439 ms_dispat           root   53u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.197 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.509 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40440 ms_local            root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:47.509 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40440 ms_local            root  DEL       REG               0,17                    200853 /[aio]
, 2022-04-03T01:40:47.510 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40440 ms_local            root  DEL       REG               0,17                    200825 /[aio]

2022-04-03T14:13:13.197 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.543 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40441 ms_dispat           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0
, 2022-04-03T01:40:47.543 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40441 ms_dispat           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0
, 2022-04-03T01:40:47.544 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40441 ms_dispat           root  mem       REG                8,1     1338272        789117 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0

2022-04-03T14:13:13.198 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.577 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40441 ms_dispat           root   46u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.578 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40441 ms_dispat           root   47u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.578 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40441 ms_dispat           root   48u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.199 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40442 ms_local            root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3
, 2022-04-03T01:40:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40442 ms_local            root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:47.612 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40442 ms_local            root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1

2022-04-03T14:13:13.199 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.646 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40442 ms_local            root   97u     IPv4             203013         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi067.front.sepia.ceph.com:42234 (ESTABLISHED)
, 2022-04-03T01:40:47.646 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40442 ms_local            root   98u     IPv4             201219         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi061.front.sepia.ceph.com:33034 (ESTABLISHED)
, 2022-04-03T01:40:47.647 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40443 ms_dispat           root  cwd       DIR                8,1        4096          4714 /home/ubuntu

2022-04-03T14:13:13.200 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.680 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40443 ms_dispat           root   41u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.680 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40443 ms_dispat           root   42u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.681 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40443 ms_dispat           root   43u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.201 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40444 ms_local            root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k
, 2022-04-03T01:40:47.714 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40444 ms_local            root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:47.715 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40444 ms_local            root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so

2022-04-03T14:13:13.201 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.748 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40444 ms_local            root   89u     IPv4             199546         0t0           TCP smithi061.front.sepia.ceph.com:36288->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:47.748 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40444 ms_local            root   90u     IPv4             223415         0t0           TCP smithi061.front.sepia.ceph.com:59036->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)
, 2022-04-03T01:40:47.748 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40444 ms_local            root   91u     IPv4             247734         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi067.front.sepia.ceph.com:42528 (ESTABLISHED)

2022-04-03T14:13:13.202 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.781 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40445 ms_dispat           root   36u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.781 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40445 ms_dispat           root   37u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:47.782 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40445 ms_dispat           root   38u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.203 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.823 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40446 ms_local            root  mem       REG                8,1      168536          8697 /usr/lib64/libselinux.so.1
, 2022-04-03T01:40:47.823 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40446 ms_local            root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0
, 2022-04-03T01:40:47.824 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40446 ms_local            root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0

2022-04-03T14:13:13.203 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.857 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40446 ms_local            root   75u     IPv4             199532         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55974 (ESTABLISHED)
, 2022-04-03T01:40:47.858 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40446 ms_local            root   76u     IPv4             199533         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi061.front.sepia.ceph.com:53402 (ESTABLISHED)
, 2022-04-03T01:40:47.858 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40446 ms_local            root   77u     IPv4             199536         0t0           TCP smithi061.front.sepia.ceph.com:33372->smithi067.front.sepia.ceph.com:6822 (ESTABLISHED)

2022-04-03T14:13:13.204 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.891 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40447 fn_anonym           root   22u     IPv4             194290         0t0           TCP *:6822 (LISTEN)
, 2022-04-03T01:40:47.892 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40447 fn_anonym           root   23u     IPv4             194315         0t0           TCP *:6824 (LISTEN)
, 2022-04-03T01:40:47.892 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40447 fn_anonym           root   24u     IPv4             192987         0t0           TCP *:6826 (LISTEN)

2022-04-03T14:13:13.205 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.931 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40448 finisher            root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:47.931 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40448 finisher            root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:47.932 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40448 finisher            root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0

2022-04-03T14:13:13.205 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:47.966 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40448 finisher            root   70u     IPv4             199527         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi061.front.sepia.ceph.com:53398 (ESTABLISHED)
, 2022-04-03T01:40:47.967 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40448 finisher            root   71u     IPv4             199528         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55902 (ESTABLISHED)
, 2022-04-03T01:40:47.967 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40448 finisher            root   72u     IPv4             199529         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi067.front.sepia.ceph.com:55920 (ESTABLISHED)

2022-04-03T14:13:13.206 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.001 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40449 safe_time           root   17w     FIFO               0,12         0t0        197774 pipe
, 2022-04-03T01:40:48.001 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40449 safe_time           root   18u     unix 0xffff88810e40c380         0t0        197775 /var/run/ceph/ceph-osd.1.asok type=STREAM
, 2022-04-03T01:40:48.002 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40449 safe_time           root   19u     IPv4             192933         0t0           TCP *:6816 (LISTEN)

2022-04-03T14:13:13.207 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.036 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40450 safe_time           root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0
, 2022-04-03T01:40:48.036 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40450 safe_time           root  mem       REG                8,1       57160        789048 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0
, 2022-04-03T01:40:48.037 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40450 safe_time           root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so

2022-04-03T14:13:13.207 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.071 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40450 safe_time           root   65u     IPv4             199520         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57858 (ESTABLISHED)
, 2022-04-03T01:40:48.071 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40450 safe_time           root   66u     IPv4             199521         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57876 (ESTABLISHED)
, 2022-04-03T01:40:48.071 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40450 safe_time           root   67u     IPv4             199522         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52982 (ESTABLISHED)

2022-04-03T14:13:13.208 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.105 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40451 ceph-osd            root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:48.106 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40451 ceph-osd            root   12r     FIFO               0,12         0t0        192876 pipe
, 2022-04-03T01:40:48.106 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40451 ceph-osd            root   13w     FIFO               0,12         0t0        192876 pipe

2022-04-03T14:13:13.209 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.140 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40452 osd_srv_a           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0
, 2022-04-03T01:40:48.140 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40452 osd_srv_a           root  mem       REG                8,1      519896        789114 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0
, 2022-04-03T01:40:48.140 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40452 osd_srv_a           root  mem       REG                8,1       53792        789105 /usr/lib64/rados-classes/libcls_numops.so.1.0.0

2022-04-03T14:13:13.209 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.174 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40452 osd_srv_a           root   60w     FIFO               0,12         0t0        196240 pipe
, 2022-04-03T01:40:48.175 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40452 osd_srv_a           root   61r     FIFO               0,12         0t0        196241 pipe
, 2022-04-03T01:40:48.175 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40452 osd_srv_a           root   62w     FIFO               0,12         0t0        196241 pipe

2022-04-03T14:13:13.210 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.209 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40453 tp_osd_tp           root    6r     FIFO               0,12         0t0        192874 pipe
, 2022-04-03T01:40:48.209 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40453 tp_osd_tp           root    7w     FIFO               0,12         0t0        192874 pipe
, 2022-04-03T01:40:48.209 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40453 tp_osd_tp           root    8u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.244 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40454 tp_osd_tp           root  mem       REG                8,1       70768        789090 /usr/lib64/rados-classes/libcls_hello.so.1.0.0
, 2022-04-03T01:40:48.244 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40454 tp_osd_tp           root  mem       REG                8,1       95480        789084 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0
, 2022-04-03T01:40:48.244 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40454 tp_osd_tp           root  mem       REG                8,1      235016        789087 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0

2022-04-03T14:13:13.211 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.279 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40454 tp_osd_tp           root   55u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.279 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40454 tp_osd_tp           root   56u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.280 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40454 tp_osd_tp           root   57u     IPv4             243184         0t0           TCP smithi061.front.sepia.ceph.com:47524->smithi067.front.sepia.ceph.com:3301 (ESTABLISHED)

2022-04-03T14:13:13.212 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.314 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40455 tp_osd_tp           root    1w     FIFO               0,12         0t0        192765 pipe
, 2022-04-03T01:40:48.314 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40455 tp_osd_tp           root    2w     FIFO               0,12         0t0        192766 pipe
, 2022-04-03T01:40:48.315 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40455 tp_osd_tp           root    3r     FIFO               0,12         0t0        192873 pipe

2022-04-03T14:13:13.213 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.355 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40456 tp_osd_tp           root  txt       REG                8,1    29386080         46353 /usr/bin/ceph-osd
, 2022-04-03T01:40:48.355 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40456 tp_osd_tp           root  mem       REG                8,1      278408        789123 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0
, 2022-04-03T01:40:48.356 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40456 tp_osd_tp           root  mem       REG                8,1       70544        789138 /usr/lib64/rados-classes/libcls_version.so.1.0.0

2022-04-03T14:13:13.213 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.389 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40456 tp_osd_tp           root   45u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.390 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40456 tp_osd_tp           root   46u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.390 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40456 tp_osd_tp           root   47u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.214 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.424 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40457 tp_osd_tp           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:48.425 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40457 tp_osd_tp           root  mem       REG                8,1      119760          9350 /usr/lib64/liblz4.so.1.8.3
, 2022-04-03T01:40:48.425 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40457 tp_osd_tp           root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8

2022-04-03T14:13:13.215 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.459 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40457 tp_osd_tp           root   95u     IPv4             221863         0t0           TCP smithi061.front.sepia.ceph.com:47454->smithi067.front.sepia.ceph.com:6810 (ESTABLISHED)
, 2022-04-03T01:40:48.459 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40457 tp_osd_tp           root   97u     IPv4             203013         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi067.front.sepia.ceph.com:42234 (ESTABLISHED)
, 2022-04-03T01:40:48.460 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40457 tp_osd_tp           root   98u     IPv4             201219         0t0           TCP smithi061.front.sepia.ceph.com:6820->smithi061.front.sepia.ceph.com:33034 (ESTABLISHED)

2022-04-03T14:13:13.215 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.493 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40458 tp_osd_tp           root   40u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.493 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40458 tp_osd_tp           root   41u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.493 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40458 tp_osd_tp           root   42u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.216 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.527 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40459 tp_osd_tp           root  mem       REG                8,1      321536          8749 /usr/lib64/libpthread-2.28.so
, 2022-04-03T01:40:48.527 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40459 tp_osd_tp           root  mem       REG                8,1     3079488         10929 /usr/lib64/libcrypto.so.1.1.1k
, 2022-04-03T01:40:48.528 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40459 tp_osd_tp           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so

2022-04-03T14:13:13.217 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.562 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40459 tp_osd_tp           root   88u     IPv4             199545         0t0           TCP smithi061.front.sepia.ceph.com:32960->smithi067.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:48.562 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40459 tp_osd_tp           root   89u     IPv4             199546         0t0           TCP smithi061.front.sepia.ceph.com:36288->smithi067.front.sepia.ceph.com:6828 (ESTABLISHED)
, 2022-04-03T01:40:48.562 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40459 tp_osd_tp           root   90u     IPv4             223415         0t0           TCP smithi061.front.sepia.ceph.com:59036->smithi067.front.sepia.ceph.com:6802 (ESTABLISHED)

2022-04-03T14:13:13.217 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.595 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40460 tp_osd_tp           root   35u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.595 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40460 tp_osd_tp           root   36u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block
, 2022-04-03T01:40:48.596 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40460 tp_osd_tp           root   37u      REG              253,1 96636764160            70 /var/lib/ceph/osd/ceph-1/block

2022-04-03T14:13:13.218 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.629 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40461 tp_osd_tp           root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so
, 2022-04-03T01:40:48.629 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40461 tp_osd_tp           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:48.629 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40461 tp_osd_tp           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2

2022-04-03T14:13:13.219 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.663 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40461 tp_osd_tp           root   83u     IPv4             201963         0t0           TCP smithi061.front.sepia.ceph.com:49372->smithi061.front.sepia.ceph.com:6814 (ESTABLISHED)
, 2022-04-03T01:40:48.663 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40461 tp_osd_tp           root   84u     IPv4             196264         0t0           TCP smithi061.front.sepia.ceph.com:35642->smithi061.front.sepia.ceph.com:6806 (ESTABLISHED)
, 2022-04-03T01:40:48.663 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40461 tp_osd_tp           root   85u     IPv4             199543         0t0           TCP smithi061.front.sepia.ceph.com:34850->smithi067.front.sepia.ceph.com:6820 (ESTABLISHED)

2022-04-03T14:13:13.219 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.697 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40462 tp_osd_tp           root   30w     FIFO               0,12         0t0        195241 pipe
, 2022-04-03T01:40:48.698 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40462 tp_osd_tp           root   31r      DIR              253,1         278            64 /var/lib/ceph/osd/ceph-1
, 2022-04-03T01:40:48.698 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40462 tp_osd_tp           root   32uW     REG              253,1          37            69 /var/lib/ceph/osd/ceph-1/fsid

2022-04-03T14:13:13.220 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40463 tp_osd_tp           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:48.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40463 tp_osd_tp           root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1
, 2022-04-03T01:40:48.736 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40463 tp_osd_tp           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0

2022-04-03T14:13:13.221 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.771 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40463 tp_osd_tp           root   78u     IPv4             199537         0t0           TCP smithi061.front.sepia.ceph.com:36816->smithi061.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:48.771 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40463 tp_osd_tp           root   79u     IPv4             199538         0t0           TCP smithi061.front.sepia.ceph.com:40150->smithi067.front.sepia.ceph.com:6804 (ESTABLISHED)
, 2022-04-03T01:40:48.771 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40463 tp_osd_tp           root   80u     IPv4             199539         0t0           TCP smithi061.front.sepia.ceph.com:60146->smithi067.front.sepia.ceph.com:6830 (ESTABLISHED)

2022-04-03T14:13:13.221 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.805 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40464 tp_osd_tp           root   25u     IPv4             194348         0t0           TCP *:6828 (LISTEN)
, 2022-04-03T01:40:48.805 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40464 tp_osd_tp           root   26u     IPv4             194380         0t0           TCP *:ambit-lm (LISTEN)
, 2022-04-03T01:40:48.805 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40464 tp_osd_tp           root   27r     FIFO               0,12         0t0        195240 pipe

2022-04-03T14:13:13.222 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.846 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40465 tp_osd_tp           root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so
, 2022-04-03T01:40:48.846 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40465 tp_osd_tp           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0
, 2022-04-03T01:40:48.846 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40465 tp_osd_tp           root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0

2022-04-03T14:13:13.223 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.879 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40465 tp_osd_tp           root   68u     IPv4             199523         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57914 (ESTABLISHED)
, 2022-04-03T01:40:48.879 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40465 tp_osd_tp           root   69u     IPv4             199524         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57926 (ESTABLISHED)
, 2022-04-03T01:40:48.880 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40465 tp_osd_tp           root   70u     IPv4             199527         0t0           TCP smithi061.front.sepia.ceph.com:6828->smithi061.front.sepia.ceph.com:53398 (ESTABLISHED)

2022-04-03T14:13:13.223 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.920 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40466 tp_osd_tp           root   14w      REG                8,1  8409854053        790593 /var/log/ceph/ceph-osd.1.log
, 2022-04-03T01:40:48.921 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40466 tp_osd_tp           root   16r     FIFO               0,12         0t0        197774 pipe
, 2022-04-03T01:40:48.921 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40466 tp_osd_tp           root   17w     FIFO               0,12         0t0        197774 pipe

2022-04-03T14:13:13.224 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.954 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40467 tp_osd_tp           root  mem       REG                8,1       74888        789081 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0
, 2022-04-03T01:40:48.954 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40467 tp_osd_tp           root  mem       REG                8,1     1874584        789120 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0
, 2022-04-03T01:40:48.955 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40467 tp_osd_tp           root  mem       REG                8,1      117344        789093 /usr/lib64/rados-classes/libcls_journal.so.1.0.0

2022-04-03T14:13:13.225 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:48.988 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40467 tp_osd_tp           root   63u     IPv4             201177         0t0           TCP smithi061.front.sepia.ceph.com:51298->smithi061.front.sepia.ceph.com:6825 (ESTABLISHED)
, 2022-04-03T01:40:48.988 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40467 tp_osd_tp           root   64u     IPv4             199519         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi061.front.sepia.ceph.com:52980 (ESTABLISHED)
, 2022-04-03T01:40:48.989 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40467 tp_osd_tp           root   65u     IPv4             199520         0t0           TCP smithi061.front.sepia.ceph.com:6824->smithi067.front.sepia.ceph.com:57858 (ESTABLISHED)

2022-04-03T14:13:13.225 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.023 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40468 tp_osd_tp           root    9r     FIFO               0,12         0t0        192875 pipe
, 2022-04-03T01:40:49.023 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40468 tp_osd_tp           root   10w     FIFO               0,12         0t0        192875 pipe
, 2022-04-03T01:40:49.024 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40468 tp_osd_tp           root   11u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.226 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40469 osd_srv_h           root  mem       REG                8,1      116704        789078 /usr/lib64/rados-classes/libcls_cas.so.1.0.0
, 2022-04-03T01:40:49.057 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40469 osd_srv_h           root  mem       REG                8,1      107640        789135 /usr/lib64/rados-classes/libcls_user.so.1.0.0
, 2022-04-03T01:40:49.058 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40469 osd_srv_h           root  mem       REG                8,1      235392        789111 /usr/lib64/rados-classes/libcls_queue.so.1.0.0

2022-04-03T14:13:13.227 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.091 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40469 osd_srv_h           root   58u     IPv4             193286         0t0           TCP smithi061.front.sepia.ceph.com:39964->smithi061.front.sepia.ceph.com:6829 (ESTABLISHED)
, 2022-04-03T01:40:49.092 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40469 osd_srv_h           root   59r     FIFO               0,12         0t0        196240 pipe
, 2022-04-03T01:40:49.092 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-osd  37054 40469 osd_srv_h           root   60w     FIFO               0,12         0t0        196240 pipe

2022-04-03T14:13:13.228 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.126 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41140                           root  mem       REG                8,1      120592         10659 /usr/lib64/libnsl.so.2.0.0
, 2022-04-03T01:40:49.126 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41140                           root  mem       REG                8,1       58088         33398 /usr/lib64/security/pam_unix.so
, 2022-04-03T01:40:49.126 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41140                           root  mem       REG                8,1       11776         31857 /usr/lib64/security/pam_localuser.so

2022-04-03T14:13:13.228 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.166 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41142                           root  mem       REG                8,1       16776          8862 /usr/lib64/libcom_err.so.2.1
, 2022-04-03T01:40:49.166 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41142                           root  mem       REG                8,1       96216         10643 /usr/lib64/libk5crypto.so.3.1
, 2022-04-03T01:40:49.166 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41142                           root  mem       REG                8,1      971456         10648 /usr/lib64/libkrb5.so.3.3

2022-04-03T14:13:13.229 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.201 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41147                           root  mem       REG                8,1       24928          9106 /usr/lib64/libcap-ng.so.0.0.0
, 2022-04-03T01:40:49.201 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41147                           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:49.201 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      41147                           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so

2022-04-03T14:13:13.230 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.235 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 41222                           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:49.236 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 41222                           root  txt       REG                8,1       11856         19642 /usr/libexec/platform-python3.6
, 2022-04-03T01:40:49.236 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 41222                           root  mem       REG                8,1       63456         40307 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so

2022-04-03T14:13:13.230 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.270 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54684 log                 root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0
, 2022-04-03T01:40:49.270 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54684 log                 root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:49.271 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54684 log                 root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0

2022-04-03T14:13:13.231 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.305 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54686 msgr-work           root   21r     FIFO               0,12         0t0        279853 pipe
, 2022-04-03T01:40:49.305 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54686 msgr-work           root   22w     FIFO               0,12         0t0        279853 pipe
, 2022-04-03T01:40:49.305 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54686 msgr-work           root   23r     FIFO               0,12         0t0        279854 pipe

2022-04-03T14:13:13.232 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.339 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54688 msgr-work           root    6r     FIFO               0,12         0t0        279846 pipe
, 2022-04-03T01:40:49.339 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54688 msgr-work           root    7w     FIFO               0,12         0t0        279846 pipe
, 2022-04-03T01:40:49.340 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54688 msgr-work           root    8u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.232 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.380 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54693 admin_soc           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11
, 2022-04-03T01:40:49.380 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54693 admin_soc           root  mem       REG                8,1      109776         16119 /usr/lib64/librdmacm.so.1.3.37.2
, 2022-04-03T01:40:49.380 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54693 admin_soc           root  mem       REG                8,1      135360         10323 /usr/lib64/libibverbs.so.1.14.37.2

2022-04-03T14:13:13.233 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.414 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54695 io_contex           root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0
, 2022-04-03T01:40:49.415 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54695 io_contex           root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:49.415 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54695 io_contex           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0

2022-04-03T14:13:13.234 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.450 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54696 io_contex           root   15uW     REG               0,24           6          1659 /run/ceph/ceph-mds.a.pid
, 2022-04-03T01:40:49.450 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54696 io_contex           root   16r     FIFO               0,12         0t0        279850 pipe
, 2022-04-03T01:40:49.451 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54696 io_contex           root   17w     FIFO               0,12         0t0        279850 pipe

2022-04-03T14:13:13.234 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.485 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54698 ms_local            root    0r      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:49.486 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54698 ms_local            root    1w     FIFO               0,12         0t0        197295 pipe
, 2022-04-03T01:40:49.486 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41229 54698 ms_local            root    2w     FIFO               0,12         0t0        197296 pipe

2022-04-03T14:13:13.235 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.520 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230                           root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so
, 2022-04-03T01:40:49.521 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230                           root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25
, 2022-04-03T01:40:49.521 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230                           root  mem       REG                8,1       95416          8875 /usr/lib64/libz.so.1.2.11

2022-04-03T14:13:13.236 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.555 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53858 msgr-work           root  txt       REG                8,1     8659672         46372 /usr/bin/ceph-mds
, 2022-04-03T01:40:49.555 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53858 msgr-work           root  mem       REG                8,1       20592         16080 /usr/lib64/liburcu-common.so.6.0.0
, 2022-04-03T01:40:49.555 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53858 msgr-work           root  mem       REG                8,1       41776         16078 /usr/lib64/liburcu-cds.so.6.0.0

2022-04-03T14:13:13.236 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.590 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53859 msgr-work           root   13w     FIFO               0,12         0t0        273707 pipe
, 2022-04-03T01:40:49.591 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53859 msgr-work           root   14w      REG                8,1     7600506        790607 /var/log/ceph/ceph-mds.c.log
, 2022-04-03T01:40:49.591 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53859 msgr-work           root   15uW     REG               0,24           6          1661 /run/ceph/ceph-mds.c.pid

2022-04-03T14:13:13.237 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.626 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53864 service             root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so
, 2022-04-03T01:40:49.626 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53864 service             root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:49.626 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53864 service             root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:13.238 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.660 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53866 signal_ha           root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:49.661 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53866 signal_ha           root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:49.661 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53866 signal_ha           root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so

2022-04-03T14:13:13.238 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.695 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53868 io_contex           root  cwd       DIR                8,1        4096          4714 /home/ubuntu
, 2022-04-03T01:40:49.695 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53868 io_contex           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:49.695 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53868 io_contex           root  txt       REG                8,1     8659672         46372 /usr/bin/ceph-mds

2022-04-03T14:13:13.239 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.729 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53869 ms_dispat           root   11u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:49.729 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53869 ms_dispat           root   12r     FIFO               0,12         0t0        273707 pipe
, 2022-04-03T01:40:49.729 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53869 ms_dispat           root   13w     FIFO               0,12         0t0        273707 pipe

2022-04-03T14:13:13.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.763 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53871 safe_time           root  mem       REG                8,1    12564712        524645 /usr/lib64/ceph/libceph-common.so.2
, 2022-04-03T01:40:49.763 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53871 safe_time           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3
, 2022-04-03T01:40:49.763 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41230 53871 safe_time           root  mem       REG                8,1       28840          8740 /usr/lib64/libdl-2.28.so

2022-04-03T14:13:13.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.799 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53009 log                 root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0
, 2022-04-03T01:40:49.799 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53009 log                 root  mem       REG                8,1       76752         39843 /usr/lib64/libunwind.so.8.0.1
, 2022-04-03T01:40:49.800 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53009 log                 root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so

2022-04-03T14:13:13.241 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.835 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53011 msgr-work           root   24w     FIFO               0,12         0t0        259052 pipe
, 2022-04-03T01:40:49.835 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53011 msgr-work           root   26u     IPv4             262614         0t0           TCP smithi061.front.sepia.ceph.com:50512->smithi061.front.sepia.ceph.com:3300 (ESTABLISHED)
, 2022-04-03T01:40:49.835 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53012 msgr-work           root  cwd       DIR                8,1        4096          4714 /home/ubuntu

2022-04-03T14:13:13.242 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53013 msgr-work           root    9r     FIFO               0,12         0t0        259045 pipe
, 2022-04-03T01:40:49.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53013 msgr-work           root   10w     FIFO               0,12         0t0        259045 pipe
, 2022-04-03T01:40:49.872 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53013 msgr-work           root   11u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.242 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.907 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53018 admin_soc           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:49.907 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53018 admin_soc           root  mem       REG                8,1      247936          9110 /usr/lib64/liblua-5.3.so
, 2022-04-03T01:40:49.908 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53018 admin_soc           root  mem       REG                8,1    12564712        524645 /usr/lib64/ceph/libceph-common.so.2

2022-04-03T14:13:13.243 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.564+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:40:49.754+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:40:49.958 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53020 io_contex           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1

2022-04-03T14:13:13.244 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:49.992 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53021 io_contex           root   16r     FIFO               0,12         0t0        259048 pipe
, 2022-04-03T01:40:49.992 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53021 io_contex           root   17w     FIFO               0,12         0t0        259048 pipe
, 2022-04-03T01:40:49.993 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53021 io_contex           root   18u     unix 0xffff88816c8be900         0t0        259049 /var/run/ceph/ceph-mds.e.asok type=STREAM

2022-04-03T14:13:13.244 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.026 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53023 ms_local            root    1w     FIFO               0,12         0t0        203068 pipe
, 2022-04-03T01:40:50.026 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53023 ms_local            root    2w     FIFO               0,12         0t0        203069 pipe
, 2022-04-03T01:40:50.027 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mds  41231 53023 ms_local            root    3r     FIFO               0,12         0t0        259043 pipe

2022-04-03T14:13:13.245 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.060 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      51994                           root  mem       REG                8,1       24744         11054 /usr/lib64/libpwquality.so.1.0.2
, 2022-04-03T01:40:50.061 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      51994                           root  mem       REG                8,1       11720         31844 /usr/lib64/security/pam_pwquality.so
, 2022-04-03T01:40:50.061 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      51994                           root  mem       REG                8,1        7672         33393 /usr/lib64/security/pam_permit.so

2022-04-03T14:13:13.246 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.095 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 52034                           root  mem       REG                8,1         167         29712 /usr/lib/locale/en_US.utf8/LC_ADDRESS
, 2022-04-03T01:40:50.096 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 52034                           root  mem       REG                8,1          59         32626 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
, 2022-04-03T01:40:50.096 INFO:teuthology.orchestra.run.smithi061.stdout:daemon-he 52034                           root  mem       REG                8,1          23         32624 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT

2022-04-03T14:13:13.246 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.130 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52040 log                 root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:50.130 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52040 log                 root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so
, 2022-04-03T01:40:50.131 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52040 log                 root  mem       REG                8,1     1661408          8921 /usr/lib64/libstdc++.so.6.0.25

2022-04-03T14:13:13.247 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.164 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52041 service             root    4w     FIFO               0,12         0t0        258326 pipe
, 2022-04-03T01:40:50.165 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52041 service             root    5w      REG                8,1    18443730        790596 /var/log/ceph/ceph-mon.a.log
, 2022-04-03T01:40:50.165 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52041 service             root    6uW     REG               0,24           6          1641 /run/ceph/ceph-mon.a.pid

2022-04-03T14:13:13.248 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.199 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52042 admin_soc           root   28u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:50.200 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52042 admin_soc           root   29r     FIFO               0,12         0t0        250842 pipe
, 2022-04-03T01:40:50.200 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52042 admin_soc           root   30w     FIFO               0,12         0t0        250842 pipe

2022-04-03T14:13:13.248 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.234 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52044 rocksdb:l           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:50.234 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52044 rocksdb:l           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0
, 2022-04-03T01:40:50.235 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52044 rocksdb:l           root  mem       REG                8,1       33464          8876 /usr/lib64/libuuid.so.1.3.0

2022-04-03T14:13:13.249 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.269 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52045 rocksdb:h           root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:50.269 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52045 rocksdb:h           root    0r      CHR                1,3         0t0             4 /dev/null
, 2022-04-03T01:40:50.270 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52045 rocksdb:h           root    1w     FIFO               0,12         0t0        256643 pipe

2022-04-03T14:13:13.250 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.304 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52061 ceph-mon            root   23r     FIFO               0,12         0t0        250840 pipe
, 2022-04-03T01:40:50.304 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52061 ceph-mon            root   24w     FIFO               0,12         0t0        250840 pipe
, 2022-04-03T01:40:50.305 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52061 ceph-mon            root   25u  a_inode               0,13           0         11317 [eventpoll]

2022-04-03T14:13:13.250 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.338 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52063 msgr-work           root  mem       REG                8,1       33776         16076 /usr/lib64/liburcu-bp.so.6.0.0
, 2022-04-03T01:40:50.338 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52063 msgr-work           root  mem       REG                8,1       66168         24193 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0
, 2022-04-03T01:40:50.339 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52063 msgr-work           root  mem       REG                8,1      543160          8693 /usr/lib64/libpcre2-8.so.0.7.1

2022-04-03T14:13:13.251 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.372 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52064 msgr-work           root  mem       REG                8,1       46256          9903 /usr/lib64/libsnappy.so.1.1.8
, 2022-04-03T01:40:50.372 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52064 msgr-work           root  mem       REG                8,1      123264          8750 /usr/lib64/libresolv-2.28.so
, 2022-04-03T01:40:50.373 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52064 msgr-work           root  mem       REG                8,1       69120          8751 /usr/lib64/librt-2.28.so

2022-04-03T14:13:13.252 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.407 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52065 msgr-work           root   18r      DIR              253,4        4096      50331782 /var/lib/ceph/mon/ceph-a/store.db
, 2022-04-03T01:40:50.407 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52065 msgr-work           root   19w      REG                8,1       68956        790600 /var/log/ceph/ceph.audit.log
, 2022-04-03T01:40:50.408 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52065 msgr-work           root   20w      REG              253,4         444      50331794 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000027

2022-04-03T14:13:13.252 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.442 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52067 fin                 root  txt       REG                8,1    11378696         46381 /usr/bin/ceph-mon
, 2022-04-03T01:40:50.442 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52067 fin                 root  mem       REG                8,1     2083840        789067 /usr/lib64/ceph/erasure-code/libec_lrc.so
, 2022-04-03T01:40:50.443 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52067 fin                 root  mem       REG                8,1      414240        789063 /usr/lib64/ceph/erasure-code/libec_jerasure.so

2022-04-03T14:13:13.253 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.477 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52068 safe_time           root  mem       REG                8,1      269368         10131 /usr/lib64/libfuse.so.2.9.7
, 2022-04-03T01:40:50.477 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52068 safe_time           root  mem       REG                8,1       35168          9347 /usr/lib64/libaio.so.1.0.1
, 2022-04-03T01:40:50.478 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52068 safe_time           root  mem       REG                8,1      334544         39853 /usr/lib64/libtcmalloc.so.4.5.3

2022-04-03T14:13:13.254 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:45.443+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:45.443398+0000)
, 2022-04-03T01:40:50.518 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52069 cpu_tp              root    9u     unix 0xffff88816c8bb480         0t0        258328 /var/run/ceph/ceph-mon.a.asok type=STREAM
, 2022-04-03T01:40:50.518 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52069 cpu_tp              root   10r     FIFO               0,12         0t0        258330 pipe

2022-04-03T14:13:13.255 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.552 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52070 cpu_tp              root   32u     IPv4             250844         0t0           TCP smithi061.front.sepia.ceph.com:smc-https (LISTEN)
, 2022-04-03T01:40:50.552 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52070 cpu_tp              root   33u     IPv4             261820         0t0           TCP smithi061.front.sepia.ceph.com:3300->smithi061.front.sepia.ceph.com:50512 (ESTABLISHED)
, 2022-04-03T01:40:50.553 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52070 cpu_tp              root   35r     FIFO               0,12         0t0        257558 pipe

2022-04-03T14:13:13.255 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.587 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52072 cpu_tp              root  mem       REG                8,1     3167104          8739 /usr/lib64/libc-2.28.so
, 2022-04-03T01:40:50.587 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52072 cpu_tp              root  mem       REG                8,1       99672         15511 /usr/lib64/libgcc_s-8-20210514.so.1
, 2022-04-03T01:40:50.588 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52072 cpu_tp              root  mem       REG                8,1     2191520          8741 /usr/lib64/libm-2.28.so

2022-04-03T14:13:13.256 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.622 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52073 ms_dispat           root    3r     FIFO               0,12         0t0        258326 pipe
, 2022-04-03T01:40:50.622 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52073 ms_dispat           root    4w     FIFO               0,12         0t0        258326 pipe
, 2022-04-03T01:40:50.623 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52073 ms_dispat           root    5w      REG                8,1    18443730        790596 /var/log/ceph/ceph-mon.a.log

2022-04-03T14:13:13.257 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.656 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52074 ms_local            root   27w     FIFO               0,12         0t0        250841 pipe
, 2022-04-03T01:40:50.657 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52074 ms_local            root   28u  a_inode               0,13           0         11317 [eventpoll]
, 2022-04-03T01:40:50.657 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52074 ms_local            root   29r     FIFO               0,12         0t0        250842 pipe

2022-04-03T14:13:13.257 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.691 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52076 ms_dispat           root  mem       REG                8,1      147064          9169 /usr/lib64/libnl-3.so.200.26.0
, 2022-04-03T01:40:50.691 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52076 ms_dispat           root  mem       REG                8,1      549568          9177 /usr/lib64/libnl-route-3.so.200.26.0
, 2022-04-03T01:40:50.691 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52076 ms_dispat           root  mem       REG                8,1      371408         10737 /usr/lib64/libmount.so.1.1.0

2022-04-03T14:13:13.258 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.726 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52077 ms_local            root  mem       REG                8,1    12564712        524645 /usr/lib64/ceph/libceph-common.so.2
, 2022-04-03T01:40:50.726 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52077 ms_local            root  mem       REG                8,1      285512          8850 /usr/lib64/ld-2.28.so
, 2022-04-03T01:40:50.727 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-mon  52039 52077 ms_local            root    0r      CHR                1,3         0t0             4 /dev/null

2022-04-03T14:13:13.259 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.760 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      52359                           root  mem       REG                8,1       11776         31857 /usr/lib64/security/pam_localuser.so
, 2022-04-03T01:40:50.761 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      52359                           root  mem       REG                8,1       11856         31875 /usr/lib64/security/pam_usertype.so
, 2022-04-03T01:40:50.761 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      52359                           root  mem       REG                8,1       11752         31851 /usr/lib64/security/pam_faildelay.so

2022-04-03T14:13:13.259 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.795 INFO:teuthology.orchestra.run.smithi061.stdout:kworker/5 53600                           root  cwd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:50.795 INFO:teuthology.orchestra.run.smithi061.stdout:kworker/5 53600                           root  rtd       DIR                8,1        4096             2 /
, 2022-04-03T01:40:50.795 INFO:teuthology.orchestra.run.smithi061.stdout:kworker/5 53600                           root  txt   unknown                                              /proc/53600/exe

2022-04-03T14:13:13.260 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.828 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      61767                           root  mem       REG                8,1       24080         33387 /usr/lib64/security/pam_limits.so
, 2022-04-03T01:40:50.828 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      61767                           root  mem       REG                8,1       11768         31856 /usr/lib64/security/pam_keyinit.so
, 2022-04-03T01:40:50.828 INFO:teuthology.orchestra.run.smithi061.stdout:sudo      61767                           root  mem       REG                8,1       46232         10565 /usr/lib64/libcrack.so.2.9.0

2022-04-03T14:13:13.261 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:lsof      61769                           root    3r      DIR               0,21           0             1 /proc
, 2022-04-03T01:40:50.861 INFO:teuthology.orchestra.run.smithi061.stdout:lsof      61769                           root    4r      DIR               0,21           0        419636 /proc/61769/fd
, 2022-04-03T01:40:50.862 INFO:teuthology.orchestra.run.smithi061.stdout:lsof      61769                           root    5w     FIFO               0,12         0t0        341883 pipe

2022-04-03T14:13:13.261 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.895 INFO:teuthology.orchestra.run.smithi061.stdout:root          98  0.0  0.0      0     0 ?        I<   Apr02   0:00  \_ [scsi_tmf_4]
, 2022-04-03T01:40:50.895 INFO:teuthology.orchestra.run.smithi061.stdout:root          99  0.0  0.0      0     0 ?        S    Apr02   0:00  \_ [scsi_eh_5]
, 2022-04-03T01:40:50.895 INFO:teuthology.orchestra.run.smithi061.stdout:root         100  0.0  0.0      0     0 ?        I<   Apr02   0:00  \_ [scsi_tmf_5]

2022-04-03T14:13:13.262 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:50.929 INFO:teuthology.orchestra.run.smithi061.stdout:root       44735  0.0  0.0      0     0 ?        I<   Apr02   0:00  \_ [ceph-msgr]
, 2022-04-03T01:40:50.930 INFO:teuthology.orchestra.run.smithi061.stdout:root       44742  0.0  0.0      0     0 ?        I<   Apr02   0:00  \_ [ceph-watch-noti]
, 2022-04-03T01:40:50.930 INFO:teuthology.orchestra.run.smithi061.stdout:root       44743  0.0  0.0      0     0 ?        I<   Apr02   0:00  \_ [ceph-completion]

2022-04-03T14:13:13.263 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:46.391+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:40:46.421+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:40:46.443+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:46.443625+0000)

2022-04-03T14:13:13.263 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:40:57.309+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:40:57.391+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:57.393094+0000)
, 2022-04-03T01:40:57.444+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:40:57.446171+0000)

2022-04-03T14:13:13.264 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:07.186+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:41:07.394+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:07.395890+0000)
, 2022-04-03T01:41:07.447+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:07.448479+0000)

2022-04-03T14:13:13.265 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:17.278+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:41:17.397+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:17.399143+0000)
, 2022-04-03T01:41:17.449+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:17.451270+0000)

2022-04-03T14:13:13.266 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:27.437+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:41:27.452+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:27.453689+0000)
, 2022-04-03T01:41:27.551+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.266 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:37.421+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:41:37.455+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:37.455872+0000)
, 2022-04-03T01:41:37.622+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:37.622588+0000)

2022-04-03T14:13:13.267 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:47.454+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:41:47.456+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:47.458041+0000)
, 2022-04-03T01:41:47.624+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:47.625495+0000)

2022-04-03T14:13:13.268 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:41:57.458+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:57.459927+0000)
, 2022-04-03T01:41:57.627+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:41:57.628541+0000)
, 2022-04-03T01:41:57.651+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.268 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:07.558+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:42:07.630+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:07.631492+0000)
, 2022-04-03T01:42:07.699+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.269 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:17.579+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:42:17.634+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:17.635208+0000)
, 2022-04-03T01:42:17.758+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.270 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:27.628+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:42:27.639+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:27.639783+0000)
, 2022-04-03T01:42:27.711+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.270 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:37.624+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:42:37.644+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:37.644846+0000)
, 2022-04-03T01:42:37.677+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.271 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:47.491+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:42:47.551+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:42:47.632+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.272 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:42:57.542+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:42:57.653+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:42:57.654076+0000)
, 2022-04-03T01:42:57.690+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.273 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:07.560+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:43:07.624+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:43:07.651+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.273 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:17.442+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:17.443015+0000)
, 2022-04-03T01:43:17.463+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:43:17.486+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:17.486845+0000)

2022-04-03T14:13:13.274 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:27.447+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:27.447421+0000)
, 2022-04-03T01:43:27.490+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:27.490406+0000)
, 2022-04-03T01:43:27.616+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.275 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:37.442+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:43:37.451+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:37.452157+0000)
, 2022-04-03T01:43:37.494+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:37.494195+0000)

2022-04-03T14:13:13.276 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:47.379+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:43:47.426+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:43:47.455+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:47.456461+0000)

2022-04-03T14:13:13.276 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:43:57.405+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:43:57.459+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:57.461067+0000)
, 2022-04-03T01:43:57.500+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:43:57.501547+0000)

2022-04-03T14:13:13.277 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:07.464+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:07.465370+0000)
, 2022-04-03T01:44:07.479+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:07.483+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.278 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:17.154+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:17.270+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:44:17.364+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.278 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:27.367+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:27.464+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:27.467+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.279 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:37.242+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:37.399+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:44:37.478+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:37.478518+0000)

2022-04-03T14:13:13.280 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:47.079+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:44:47.080945+0000)
, 2022-04-03T01:44:47.130+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:47.324+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.281 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:44:57.260+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:57.286+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:44:57.442+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.281 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:07.088+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:07.089231+0000)
, 2022-04-03T01:45:07.287+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:45:07.294+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.282 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:16.790+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:45:16.963+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:45:17.093+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:17.093750+0000)

2022-04-03T14:13:13.283 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:27.012+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:45:27.097+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:27.097973+0000)
, 2022-04-03T01:45:27.270+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.283 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:36.936+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:45:37.100+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:37.101998+0000)
, 2022-04-03T01:45:37.219+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.284 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:46.682+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:45:46.729+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:45:46.730398+0000)
, 2022-04-03T01:45:46.938+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.285 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:45:56.774+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:45:56.817+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:45:57.069+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.285 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:06.738+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:06.739082+0000)
, 2022-04-03T01:46:06.780+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:46:07.114+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:07.115133+0000)

2022-04-03T14:13:13.286 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:16.592+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:46:16.678+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:46:16.694+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.287 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:26.673+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:46:26.749+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:26.749494+0000)
, 2022-04-03T01:46:26.825+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.288 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:36.753+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:36.754457+0000)
, 2022-04-03T01:46:36.777+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:46:36.799+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.288 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:46.611+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:46:46.746+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:46:46.757+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:46.758668+0000)

2022-04-03T14:13:13.289 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:46:56.762+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:46:56.763249+0000)
, 2022-04-03T01:46:56.848+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:46:56.913+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.290 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:06.767+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:06.768079+0000)
, 2022-04-03T01:47:06.819+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:47:06.854+0000 7fa9c620e700 -1 osd.4 75 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.291 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:16.549+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:16.549623+0000)
, 2022-04-03T01:47:16.575+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:16.576049+0000)
, 2022-04-03T01:47:16.772+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:16.772510+0000)

2022-04-03T14:13:13.291 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:26.579+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:26.579705+0000)
, 2022-04-03T01:47:26.776+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:26.776971+0000)
, 2022-04-03T01:47:26.819+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.292 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:36.582+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:36.583350+0000)
, 2022-04-03T01:47:36.771+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:47:36.780+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:36.781411+0000)

2022-04-03T14:13:13.293 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:46.485+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:47:46.563+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:46.564058+0000)
, 2022-04-03T01:47:46.586+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:46.587691+0000)

2022-04-03T14:13:13.293 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:47:55.706+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:47:55.788+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:47:55.789737+0000)
, 2022-04-03T01:47:56.011+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.294 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:05.747+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:48:05.793+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:05.794779+0000)
, 2022-04-03T01:48:05.999+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.295 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:15.597+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:15.597859+0000)
, 2022-04-03T01:48:15.653+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:48:15.799+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:15.799370+0000)

2022-04-03T14:13:13.295 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:25.600+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:25.600851+0000)
, 2022-04-03T01:48:25.633+0000 7f7b589b7700 -1 osd.5 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:48:25.803+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:25.803912+0000)

2022-04-03T14:13:13.296 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:35.603+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:35.604681+0000)
, 2022-04-03T01:48:35.704+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:48:35.807+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:35.808842+0000)

2022-04-03T14:13:13.297 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:45.607+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:45.607965+0000)
, 2022-04-03T01:48:45.647+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:48:45.696+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.298 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:48:55.610+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:48:55.611581+0000)
, 2022-04-03T01:48:55.686+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:48:55.699+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.298 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:05.598+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:05.599918+0000)
, 2022-04-03T01:49:05.614+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:05.615148+0000)
, 2022-04-03T01:49:05.762+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.299 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:15.586+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:49:15.603+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:15.604138+0000)
, 2022-04-03T01:49:15.618+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:15.618379+0000)

2022-04-03T14:13:13.300 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:25.600+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:49:25.608+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:25.609202+0000)
, 2022-04-03T01:49:25.621+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:25.621870+0000)

2022-04-03T14:13:13.300 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:35.612+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:35.613679+0000)
, 2022-04-03T01:49:35.624+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:35.625485+0000)
, 2022-04-03T01:49:35.640+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.301 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:45.326+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:49:45.617+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:45.618315+0000)
, 2022-04-03T01:49:45.627+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:45.628608+0000)

2022-04-03T14:13:13.302 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:49:55.483+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:49:55.496+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:49:55.622+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:49:55.622952+0000)

2022-04-03T14:13:13.303 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:05.373+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:50:05.514+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:50:05.627+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:05.627586+0000)

2022-04-03T14:13:13.304 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:15.356+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:50:15.414+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:50:15.471+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.304 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:25.257+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:50:25.443+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:50:25.592+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.305 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:35.275+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:50:35.479+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:50:35.635+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.306 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:45.318 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:50:45.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:50:45.437+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:50:55.237+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:50:55.238440+0000)
, 2022-04-03T01:50:55.310+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:50:55.357+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:05.193+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:51:05.242+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:05.242761+0000)
, 2022-04-03T01:51:05.292+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.308 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:15.246+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:15.246863+0000)
, 2022-04-03T01:51:15.399 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T01:51:15.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.309 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:25.202+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:51:25.204+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:51:25.250+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:25.251090+0000)

2022-04-03T14:13:13.309 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:35.253+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:35.255175+0000)
, 2022-04-03T01:51:35.295+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:51:35.373+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.310 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:45.330+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:51:45.439+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:51:45.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.311 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:51:54.900+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:54.900997+0000)
, 2022-04-03T01:51:55.263+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:51:55.263786+0000)
, 2022-04-03T01:51:55.348+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:04.905+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:04.906000+0000)
, 2022-04-03T01:52:05.267+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:05.267923+0000)
, 2022-04-03T01:52:05.518+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:15.271+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:15.272127+0000)
, 2022-04-03T01:52:15.491+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:52:15.516+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.313 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:24.686+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:24.688249+0000)
, 2022-04-03T01:52:24.914+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:24.915354+0000)
, 2022-04-03T01:52:25.275+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:25.276846+0000)

2022-04-03T14:13:13.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:34.879+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:52:34.918+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:34.919932+0000)
, 2022-04-03T01:52:35.280+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:35.281568+0000)

2022-04-03T14:13:13.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:44.923+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:44.924242+0000)
, 2022-04-03T01:52:45.243+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:52:45.284+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:45.286106+0000)

2022-04-03T14:13:13.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:52:54.701+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:54.701793+0000)
, 2022-04-03T01:52:54.903+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:52:54.928+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:52:54.928951+0000)

2022-04-03T14:13:13.316 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:04.705+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:04.706064+0000)
, 2022-04-03T01:53:04.899+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:53:04.933+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:04.933596+0000)

2022-04-03T14:13:13.316 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:14.938+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:14.938633+0000)
, 2022-04-03T01:53:15.013+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:53:15.209+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:24.709+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:24.710560+0000)
, 2022-04-03T01:53:24.713+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:24.715248+0000)
, 2022-04-03T01:53:24.942+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:24.943491+0000)

2022-04-03T14:13:13.318 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:34.712+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:34.714235+0000)
, 2022-04-03T01:53:34.718+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:34.719618+0000)
, 2022-04-03T01:53:34.947+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:34.948057+0000)

2022-04-03T14:13:13.319 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:44.723+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:44.724285+0000)
, 2022-04-03T01:53:44.923+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:53:44.952+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:44.953104+0000)

2022-04-03T14:13:13.319 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:53:54.643+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:53:54.721+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:54.722014+0000)
, 2022-04-03T01:53:54.728+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:53:54.729132+0000)

2022-04-03T14:13:13.320 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:04.631+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:04.725+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:04.725833+0000)
, 2022-04-03T01:54:04.733+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:04.733566+0000)

2022-04-03T14:13:13.321 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:14.738+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:14.738238+0000)
, 2022-04-03T01:54:14.829+0000 7f7b589b7700 -1 osd.5 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:14.849+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.321 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:24.642+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:24.732+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:24.733148+0000)
, 2022-04-03T01:54:24.741+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:24.742907+0000)

2022-04-03T14:13:13.322 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:34.653+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:34.717+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:34.736+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:34.737106+0000)

2022-04-03T14:13:13.323 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:44.737+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:44.739+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:44.740915+0000)
, 2022-04-03T01:54:44.751+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:54:44.752391+0000)

2022-04-03T14:13:13.324 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:54:54.462+0000 7fa9c620e700 -1 osd.4 76 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:54.516+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:54:54.634+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.324 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:03.344+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:55:03.345063+0000)
, 2022-04-03T01:55:03.368+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 35 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:55:03.448+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.325 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:13.325+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:55:13.348+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:55:13.349365+0000)
, 2022-04-03T01:55:13.675+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.326 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:23.217+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:55:23.245+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:55:23.352+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:55:23.353720+0000)

2022-04-03T14:13:13.326 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:33.213+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:55:33.262+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:55:33.302+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.327 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:43.119+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:55:43.283+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:55:43.286+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.328 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read
,     return self.channel.recv_stderr(size)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr

2022-04-03T14:13:13.330 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:55:58.188+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:55:58.214+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:55:58.262+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.330 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:08.076+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:56:08.273+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:56:08.340+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:17.795+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:17.796718+0000)
, 2022-04-03T01:56:18.024+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:18.025704+0000)
, 2022-04-03T01:56:18.135+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.332 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:28.029+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:28.030537+0000)
, 2022-04-03T01:56:28.146+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:56:28.235+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.333 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:38.034+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:38.035250+0000)
, 2022-04-03T01:56:38.280+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 33 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:56:38.323+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:47.809+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:47.810457+0000)
, 2022-04-03T01:56:47.942+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:56:48.039+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:48.040061+0000)

2022-04-03T14:13:13.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:56:57.965+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:56:58.044+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:56:58.044912+0000)
, 2022-04-03T01:56:58.140+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:07.900+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:57:08.049+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:08.049967+0000)
, 2022-04-03T01:57:08.206+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:17.801+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:17.802690+0000)
, 2022-04-03T01:57:17.823+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:17.824032+0000)
, 2022-04-03T01:57:17.829+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.337 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:27.805+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:27.806380+0000)
, 2022-04-03T01:57:27.827+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:27.828609+0000)
, 2022-04-03T01:57:28.058+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:28.059497+0000)

2022-04-03T14:13:13.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:37.808+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:37.810045+0000)
, 2022-04-03T01:57:37.832+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:37.833412+0000)
, 2022-04-03T01:57:38.063+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:38.064272+0000)

2022-04-03T14:13:13.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:47.723+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:57:47.751+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:57:47.813+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:47.813816+0000)

2022-04-03T14:13:13.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:57:57.816+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:57.817143+0000)
, 2022-04-03T01:57:57.842+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:57:57.842690+0000)
, 2022-04-03T01:57:57.872+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.340 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:07.819+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:07.820781+0000)
, 2022-04-03T01:58:07.845+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:07.847193+0000)
, 2022-04-03T01:58:07.865+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:17.538+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:17.803+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:17.822+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:17.824215+0000)

2022-04-03T14:13:13.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:27.650+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:27.827+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:27.828282+0000)
, 2022-04-03T01:58:27.855+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:27.856101+0000)

2022-04-03T14:13:13.342 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:37.681+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:37.831+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:37.832261+0000)
, 2022-04-03T01:58:37.859+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:37.860463+0000)

2022-04-03T14:13:13.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:47.391+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:47.443+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:47.443368+0000)
, 2022-04-03T01:58:47.644+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:58:57.567+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:57.751+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:58:57.839+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:58:57.839683+0000)

2022-04-03T14:13:13.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:07.655+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:59:07.760+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:59:07.842+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:07.843401+0000)

2022-04-03T14:13:13.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:17.145+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T01:59:17.456+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:17.457391+0000)
, 2022-04-03T01:59:17.745+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:27.461+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:27.461824+0000)
, 2022-04-03T01:59:27.827+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:59:27.850+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.346 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:37.465+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:37.466134+0000)
, 2022-04-03T01:59:37.751+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:59:37.853+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:37.854304+0000)

2022-04-03T14:13:13.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:47.146+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T01:59:47.175+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:59:47.470+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:47.470535+0000)

2022-04-03T14:13:13.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T01:59:57.184+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T01:59:57.475+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T00:59:57.475339+0000)
, 2022-04-03T01:59:57.604+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:07.222+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:00:07.478+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:07.479801+0000)
, 2022-04-03T02:00:07.607+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:17.131+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:00:17.135+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:17.136502+0000)
, 2022-04-03T02:00:17.241+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:27.178+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:00:27.279+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:00:27.451+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:37.250+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:00:37.330+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:00:37.450+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:47.058+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:00:47.150+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:47.150664+0000)
, 2022-04-03T02:00:47.197+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:00:57.155+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:00:57.155202+0000)
, 2022-04-03T02:00:57.171+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:00:57.420+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:07.231+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:01:07.250+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:07.388+0000 7f7b589b7700 -1 osd.5 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:16.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:01:17.163+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:17.164497+0000)
, 2022-04-03T02:01:17.174+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.354 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:27.168+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:27.169142+0000)
, 2022-04-03T02:01:27.216+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:27.277+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:37.107+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:37.173+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:37.173708+0000)
, 2022-04-03T02:01:37.379+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:47.024 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:01:47.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:01:47.164+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:01:56.952+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:56.953400+0000)
, 2022-04-03T02:01:57.150+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:01:57.181+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:01:57.182954+0000)

2022-04-03T14:13:13.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:06.957+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:06.957967+0000)
, 2022-04-03T02:02:07.024+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:02:07.187+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:07.187900+0000)

2022-04-03T14:13:13.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:16.961+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:16.962789+0000)
, 2022-04-03T02:02:17.089+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:02:17.105 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.359 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:26.920+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:26.920801+0000)
, 2022-04-03T02:02:26.966+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:26.967524+0000)
, 2022-04-03T02:02:27.045+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:36.924+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:36.924688+0000)
, 2022-04-03T02:02:36.971+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:36.972022+0000)
, 2022-04-03T02:02:37.178+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:46.928+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:46.928558+0000)
, 2022-04-03T02:02:46.976+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:46.977120+0000)
, 2022-04-03T02:02:47.160+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.361 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:02:56.881+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:02:56.930+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:56.932103+0000)
, 2022-04-03T02:02:56.980+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:02:56.981667+0000)

2022-04-03T14:13:13.362 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:06.201+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:06.215+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:06.215910+0000)
, 2022-04-03T02:03:06.273+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:16.219+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:16.220673+0000)
, 2022-04-03T02:03:16.264+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:16.269+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:26.061+0000 7fa9c620e700 -1 osd.4 78 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:03:26.168+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:26.225+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:26.225544+0000)

2022-04-03T14:13:13.364 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:36.143+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:36.143+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:03:36.229+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:36.229841+0000)

2022-04-03T14:13:13.365 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:46.140+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:03:46.189+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:03:46.234+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:46.234778+0000)

2022-04-03T14:13:13.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:03:55.987+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:03:56.008+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:03:56.009712+0000)
, 2022-04-03T02:03:56.185+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:06.001+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:06.013+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:06.014797+0000)
, 2022-04-03T02:04:06.081+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.367 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:16.019+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:16.019711+0000)
, 2022-04-03T02:04:16.143+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:04:16.178+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:25.860+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:25.967+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:25.967657+0000)
, 2022-04-03T02:04:26.023+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:26.024328+0000)

2022-04-03T14:13:13.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:35.816+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:35.955+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:35.971+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:35.971342+0000)

2022-04-03T14:13:13.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:45.957+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:45.974+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:45.974981+0000)
, 2022-04-03T02:04:46.033+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:46.033559+0000)

2022-04-03T14:13:13.370 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:04:55.635+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:55.861+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:04:55.977+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:04:55.978727+0000)

2022-04-03T14:13:13.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:05.613+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:05.614283+0000)
, 2022-04-03T02:05:05.981+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:05.982485+0000)
, 2022-04-03T02:05:06.042+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:06.043317+0000)

2022-04-03T14:13:13.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:15.985+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:15.986683+0000)
, 2022-04-03T02:05:16.016+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:05:16.047+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:16.047918+0000)

2022-04-03T14:13:13.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:25.622+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:25.623153+0000)
, 2022-04-03T02:05:25.688+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:05:25.988+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:35.627+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:35.627754+0000)
, 2022-04-03T02:05:35.644+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:05:35.994+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:35.994997+0000)

2022-04-03T14:13:13.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:45.658+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:05:45.999+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:45.999247+0000)
, 2022-04-03T02:05:46.032+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:05:55.142+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:05:55.294+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:05:55.295411+0000)
, 2022-04-03T02:05:55.295+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:05.069+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:06:05.249+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:06:05.297+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:05.298685+0000)

2022-04-03T14:13:13.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:15.151+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:06:15.300+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:15.301652+0000)
, 2022-04-03T02:06:15.606+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:25.177+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:06:25.303+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:25.304903+0000)
, 2022-04-03T02:06:25.598+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:35.221+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:06:35.308+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:35.308464+0000)
, 2022-04-03T02:06:35.527+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:45.310+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:45.311659+0000)
, 2022-04-03T02:06:45.513+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:06:45.572+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:06:55.313+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:55.314743+0000)
, 2022-04-03T02:06:55.547+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:06:55.652+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:06:55.654065+0000)

2022-04-03T14:13:13.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:05.316+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:05.317897+0000)
, 2022-04-03T02:07:05.531+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:07:05.656+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:05.657346+0000)

2022-04-03T14:13:13.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:15.610+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:07:15.638+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:07:15.659+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:15.660573+0000)

2022-04-03T14:13:13.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:25.645+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:07:25.663+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:25.663952+0000)
, 2022-04-03T02:07:25.671+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:35.404+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:07:35.667+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:35.668261+0000)
, 2022-04-03T02:07:35.810+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:45.671+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:45.672894+0000)
, 2022-04-03T02:07:45.848+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:07:45.928+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.383 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:07:55.562 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:07:55.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:07:55.676+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:07:55.677345+0000)

2022-04-03T14:13:13.384 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:05.342+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:05.343377+0000)
, 2022-04-03T02:08:05.552+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:08:05.681+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:05.682195+0000)

2022-04-03T14:13:13.385 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:15.557+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:08:15.686+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:15.687234+0000)
, 2022-04-03T02:08:15.921+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.385 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:25.564+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:08:25.632 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:08:25.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:35.143+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:08:35.358+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:35.358225+0000)
, 2022-04-03T02:08:35.562+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.387 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:45.361+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:45.362997+0000)
, 2022-04-03T02:08:45.553+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:08:45.699+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:45.701025+0000)

2022-04-03T14:13:13.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:08:55.366+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:08:55.367477+0000)
, 2022-04-03T02:08:55.542+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:08:55.557+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:05.141+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:05.142298+0000)
, 2022-04-03T02:09:05.226+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:09:05.371+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:05.372432+0000)

2022-04-03T14:13:13.389 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:15.200+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:09:15.203+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:09:15.376+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:15.377288+0000)

2022-04-03T14:13:13.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:25.243+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:09:25.382+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:25.382305+0000)
, 2022-04-03T02:09:25.405+0000 7fa9c620e700 -1 osd.4 79 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:35.156+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:35.156310+0000)
, 2022-04-03T02:09:35.229+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:09:35.239+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.391 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:45.266+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:09:45.306+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:09:45.343+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.392 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:09:55.164+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:09:55.165734+0000)
, 2022-04-03T02:09:55.222+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:09:55.366+0000 7f7b589b7700 -1 osd.5 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:04.132+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:10:04.169+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:04.170330+0000)
, 2022-04-03T02:10:04.217+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:14.131+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:10:14.174+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:14.174832+0000)
, 2022-04-03T02:10:14.237+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.394 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:24.144+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:24.170+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:10:24.179+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:24.180117+0000)

2022-04-03T14:13:13.395 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:34.050+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:10:34.071+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:10:34.097+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:34.097858+0000)

2022-04-03T14:13:13.396 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:44.057+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:10:44.101+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:44.102059+0000)
, 2022-04-03T02:10:44.154+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.396 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:10:53.965+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:10:53.980+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:10:54.104+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:10:54.105619+0000)

2022-04-03T14:13:13.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:03.783+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:03.832+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:03.915+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:13.865+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:13.962+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:11:14.015+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:23.895+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:23.915+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:24.020+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.399 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:33.600+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:33.773+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:33.774757+0000)
, 2022-04-03T02:11:33.905+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:43.778+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:43.779028+0000)
, 2022-04-03T02:11:43.825+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:44.053+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.401 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:11:53.739+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:11:53.782+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:11:53.783731+0000)
, 2022-04-03T02:11:53.908+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.401 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:03.458+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:03.459265+0000)
, 2022-04-03T02:12:03.609+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:03.771+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.402 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:13.658+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:13.670+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:13.792+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:13.792782+0000)

2022-04-03T14:13:13.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:23.622+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:23.627+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:23.797+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:23.797462+0000)

2022-04-03T14:13:13.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:33.365+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:33.473+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:33.474591+0000)
, 2022-04-03T02:12:33.586+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:43.478+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:43.479084+0000)
, 2022-04-03T02:12:43.495+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:43.731+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:12:53.449+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:12:53.483+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:53.484208+0000)
, 2022-04-03T02:12:53.811+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:12:53.811944+0000)

2022-04-03T14:13:13.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:03.254+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:03.255865+0000)
, 2022-04-03T02:13:03.487+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:03.488729+0000)
, 2022-04-03T02:13:03.507+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:13.406+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:13:13.493+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:13.493779+0000)
, 2022-04-03T02:13:13.508+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.407 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:23.498+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:23.498443+0000)
, 2022-04-03T02:13:23.519+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:13:23.592+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.408 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:33.166+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:33.167871+0000)
, 2022-04-03T02:13:33.269+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:33.270416+0000)
, 2022-04-03T02:13:33.461+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.409 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:43.274+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:43.275330+0000)
, 2022-04-03T02:13:43.491+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:13:43.507+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:43.507846+0000)

2022-04-03T14:13:13.409 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:13:53.279+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:53.280400+0000)
, 2022-04-03T02:13:53.460+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:13:53.512+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:13:53.512786+0000)

2022-04-03T14:13:13.410 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:03.040+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:14:03.179+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:03.179445+0000)
, 2022-04-03T02:14:03.284+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:03.285434+0000)

2022-04-03T14:13:13.411 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:13.182+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:13.183144+0000)
, 2022-04-03T02:14:13.290+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:13.290623+0000)
, 2022-04-03T02:14:13.468+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.411 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:23.186+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:23.187009+0000)
, 2022-04-03T02:14:23.295+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:23.295346+0000)
, 2022-04-03T02:14:23.527+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:23.527319+0000)

2022-04-03T14:13:13.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:33.029+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:14:33.083+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:14:33.189+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:33.190725+0000)

2022-04-03T14:13:13.413 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:43.136+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:14:43.193+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:43.194542+0000)
, 2022-04-03T02:14:43.304+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:43.305306+0000)

2022-04-03T14:13:13.414 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:14:53.180+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:14:53.198+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:53.198916+0000)
, 2022-04-03T02:14:53.309+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:14:53.310355+0000)

2022-04-03T14:13:13.414 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:03.194+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:03.202+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:03.202914+0000)
, 2022-04-03T02:15:03.259+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.415 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:13.206+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:13.206962+0000)
, 2022-04-03T02:15:13.256+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:15:13.320+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:13.320387+0000)

2022-04-03T14:13:13.416 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:23.209+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:23.210912+0000)
, 2022-04-03T02:15:23.295+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:15:23.324+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:23.325469+0000)

2022-04-03T14:13:13.417 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:33.045+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:15:33.099+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:15:33.213+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:33.214650+0000)

2022-04-03T14:13:13.417 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:43.081+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:15:43.218+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:43.218781+0000)
, 2022-04-03T02:15:43.242+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.418 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:15:53.009+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:15:53.193+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:15:53.221+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:15:53.222522+0000)

2022-04-03T14:13:13.419 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:02.853+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:16:02.899+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:02.899685+0000)
, 2022-04-03T02:16:03.045+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.420 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:12.903+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:12.904201+0000)
, 2022-04-03T02:16:13.035+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:16:13.213+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.421 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:22.907+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:22.908776+0000)
, 2022-04-03T02:16:22.992+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:16:23.233+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:23.234822+0000)

2022-04-03T14:13:13.423 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:32.604+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:16:32.892+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:16:32.912+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:32.913324+0000)

2022-04-03T14:13:13.424 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:42.917+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:42.918396+0000)
, 2022-04-03T02:16:42.962+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:16:43.078+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.425 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:16:52.922+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:16:52.922993+0000)
, 2022-04-03T02:16:53.096+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:16:53.142+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.426 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:02.601+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:02.601728+0000)
, 2022-04-03T02:17:02.681+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:17:02.927+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:02.927641+0000)

2022-04-03T14:13:13.427 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:12.606+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:12.607077+0000)
, 2022-04-03T02:17:12.931+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:12.932186+0000)
, 2022-04-03T02:17:13.078+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.428 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:22.681+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:17:22.936+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:22.937067+0000)
, 2022-04-03T02:17:23.107+0000 7f7b589b7700 -1 osd.5 81 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.428 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:32.403+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:17:32.616+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:32.617142+0000)
, 2022-04-03T02:17:32.659+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.429 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:42.621+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:42.621989+0000)
, 2022-04-03T02:17:42.621+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:17:42.944+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:42.946129+0000)

2022-04-03T14:13:13.430 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:17:52.626+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:52.626902+0000)
, 2022-04-03T02:17:52.733+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:17:52.950+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:17:52.951007+0000)

2022-04-03T14:13:13.431 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:02.401+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:02.402170+0000)
, 2022-04-03T02:18:02.494+0000 7fa9c620e700 -1 osd.4 81 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:18:02.622+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.431 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:12.220+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:18:12.243+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:18:12.278+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:12.278739+0000)

2022-04-03T14:13:13.432 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:22.205+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:18:22.281+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:22.282635+0000)
, 2022-04-03T02:18:22.341+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.433 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:31.968+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:18:31.969361+0000)
, 2022-04-03T02:18:32.103+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:18:32.170+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.434 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:42.092+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:18:42.210+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:18:42.242+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.434 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:18:52.133+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:18:52.155+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:18:52.220+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.435 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:01.818+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:19:01.982+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:01.982889+0000)
, 2022-04-03T02:19:02.116+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.436 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:11.965+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:19:11.987+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:11.987575+0000)
, 2022-04-03T02:19:12.132+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.437 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:21.998+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:19:22.048+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:19:22.127+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.437 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:31.996+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:31.997088+0000)
, 2022-04-03T02:19:32.065+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:19:32.070+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.438 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:42.001+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:42.001651+0000)
, 2022-04-03T02:19:42.043+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:19:42.074+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.439 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:19:52.006+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:19:52.006412+0000)
, 2022-04-03T02:19:52.046+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:19:52.050+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.440 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:01.689+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:01.689422+0000)
, 2022-04-03T02:20:01.874+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:20:02.011+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:02.011368+0000)

2022-04-03T14:13:13.440 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:11.823+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:20:12.014+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:12.015898+0000)
, 2022-04-03T02:20:12.062+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.441 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:21.784+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:20:22.019+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:22.020859+0000)
, 2022-04-03T02:20:22.135+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.442 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:31.537+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:20:31.703+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:31.704393+0000)
, 2022-04-03T02:20:31.738+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.442 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:41.709+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:41.709836+0000)
, 2022-04-03T02:20:41.776+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:20:41.999+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.443 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:20:51.714+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:20:51.714727+0000)
, 2022-04-03T02:20:51.816+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:20:52.006+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.444 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:01.488+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:01.489052+0000)
, 2022-04-03T02:21:01.570+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:21:01.719+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:01.719649+0000)

2022-04-03T14:13:13.444 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:11.523+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:21:11.723+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:11.724605+0000)
, 2022-04-03T02:21:11.933+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.445 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:21.650+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:21:21.728+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:21.729639+0000)
, 2022-04-03T02:21:22.009+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.446 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:31.493+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:31.503+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:31.504284+0000)
, 2022-04-03T02:21:31.659+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.447 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:41.591+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:41.617+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:21:41.739+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:41.739686+0000)

2022-04-03T14:13:13.447 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:21:51.514+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:21:51.698+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:21:51.744+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:21:51.744524+0000)

2022-04-03T14:13:13.448 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:01.370+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:01.371018+0000)
, 2022-04-03T02:22:01.442+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:22:01.518+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:01.519259+0000)

2022-04-03T14:13:13.449 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:11.432+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:22:11.523+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:11.524196+0000)
, 2022-04-03T02:22:11.673+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.449 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:21.528+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:21.528964+0000)
, 2022-04-03T02:22:21.558+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:22:21.667+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.450 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:31.282+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:22:31.382+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:31.383698+0000)
, 2022-04-03T02:22:31.513+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.451 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:41.436+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:22:41.524+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:22:41.538+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:41.539269+0000)

2022-04-03T14:13:13.452 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:22:51.480+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:22:51.486+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:22:51.544+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:22:51.544533+0000)

2022-04-03T14:13:13.452 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:01.209+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:23:01.395+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:01.395569+0000)
, 2022-04-03T02:23:01.470+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.453 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:11.399+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:11.400264+0000)
, 2022-04-03T02:23:11.512+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:23:11.553+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:11.554756+0000)

2022-04-03T14:13:13.454 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:21.403+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:21.404405+0000)
, 2022-04-03T02:23:21.515+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:23:21.523+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.455 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:31.256+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:23:31.274+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:23:31.296+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.456 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:41.297+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:23:41.307+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:23:41.412+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:41.412641+0000)

2022-04-03T14:13:13.456 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:23:51.309+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:23:51.415+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:23:51.416212+0000)
, 2022-04-03T02:23:51.463+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.457 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:00.808+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:00.808346+0000)
, 2022-04-03T02:24:00.921+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:24:00.985+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.458 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:10.888+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:24:10.988+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:24:11.059+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.459 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:20.813+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:20.814494+0000)
, 2022-04-03T02:24:21.003+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:24:21.127+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.459 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:31.028+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:24:31.131+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:31.132991+0000)
, 2022-04-03T02:24:31.205+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.460 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:40.977+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:24:41.135+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:41.136480+0000)
, 2022-04-03T02:24:41.224+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.461 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:24:50.937+0000 7fa9c620e700 -1 osd.4 82 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:24:51.139+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:24:51.139478+0000)
, 2022-04-03T02:24:51.253+0000 7f7b589b7700 -1 osd.5 82 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.462 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:01.141+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:01.142430+0000)
, 2022-04-03T02:25:01.214+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:25:01.304+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.463 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:10.144+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:10.145737+0000)
, 2022-04-03T02:25:10.320+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:10.373+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.463 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:20.147+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:20.148572+0000)
, 2022-04-03T02:25:20.390+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:20.434+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.464 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:30.309+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:30.417+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:30.443+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:30.444366+0000)

2022-04-03T14:13:13.465 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:40.379+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:40.407+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:40.446+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:40.447365+0000)

2022-04-03T14:13:13.466 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:25:50.227+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:50.388+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:25:50.450+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:25:50.451190+0000)

2022-04-03T14:13:13.466 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:00.454+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:00.455394+0000)
, 2022-04-03T02:26:00.560+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:26:00.586+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.467 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:10.167+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:10.169118+0000)
, 2022-04-03T02:26:10.340+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:26:10.417+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.468 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:20.255+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:26:20.353+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:26:20.363+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.469 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:30.187+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:26:30.371+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:26:30.467+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:30.468061+0000)

2022-04-03T14:13:13.469 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:40.182+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:40.183055+0000)
, 2022-04-03T02:26:40.212+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:26:40.306+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.470 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:26:50.187+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:26:50.188114+0000)
, 2022-04-03T02:26:50.322+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:26:50.393+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.471 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:00.391+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:27:00.399+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:27:00.428+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.471 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:09.908+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:27:10.196+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:10.197760+0000)
, 2022-04-03T02:27:10.329+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.472 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:19.954+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:27:20.201+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:20.202342+0000)
, 2022-04-03T02:27:20.245+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.473 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:30.206+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:30.206758+0000)
, 2022-04-03T02:27:30.253+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:27:30.313+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.474 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:39.908+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:39.908348+0000)
, 2022-04-03T02:27:40.185+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:27:40.210+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:40.211097+0000)

2022-04-03T14:13:13.474 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:27:49.913+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:27:49.913283+0000)
, 2022-04-03T02:27:50.136+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:27:50.202+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.475 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:00.040+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:00.104+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:00.219+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:00.220274+0000)

2022-04-03T14:13:13.476 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:09.778+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:09.922+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:09.923180+0000)
, 2022-04-03T02:28:10.034+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.476 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:19.771+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:19.927+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:19.928440+0000)
, 2022-04-03T02:28:20.014+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.477 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:29.932+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:29.933285+0000)
, 2022-04-03T02:28:30.053+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:30.198+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.478 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:39.736+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:28:39.827+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:39.938+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:39.938258+0000)

2022-04-03T14:13:13.479 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:49.704+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:49.704941+0000)
, 2022-04-03T02:28:49.801+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:49.880+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.479 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:28:59.708+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:59.709942+0000)
, 2022-04-03T02:28:59.781+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:28:59.946+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:28:59.947615+0000)

2022-04-03T14:13:13.480 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:09.531+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:09.532433+0000)
, 2022-04-03T02:29:09.584+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:29:09.714+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:09.714795+0000)

2022-04-03T14:13:13.481 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:19.535+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:19.536672+0000)
, 2022-04-03T02:29:19.593+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:19.719+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:19.719823+0000)

2022-04-03T14:13:13.482 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:29.559+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:29.723+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:29.724865+0000)
, 2022-04-03T02:29:29.730+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.482 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:39.455+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:29:39.495+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:39.544+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:39.544887+0000)

2022-04-03T14:13:13.483 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:49.486+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:29:49.515+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:29:49.548+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:49.549392+0000)

2022-04-03T14:13:13.484 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:29:59.440+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:29:59.551+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:29:59.553221+0000)
, 2022-04-03T02:29:59.696+0000 7fa9c620e700 -1 osd.4 83 get_health_metrics reporting 37 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.484 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:09.424+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:09.482+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:09.487+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.485 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:19.381+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:30:19.426+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:19.461+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.486 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:29.459+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:29.563+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:29.565+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:29.566299+0000)

2022-04-03T14:13:13.487 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:39.308+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:39.408+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:30:39.459+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.487 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:49.341+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:49.397+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:49.569+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.488 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:30:59.572+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:30:59.577+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:30:59.579121+0000)
, 2022-04-03T02:30:59.602+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.489 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:09.161+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:31:09.306+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:09.307389+0000)
, 2022-04-03T02:31:09.582+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:09.583405+0000)

2022-04-03T14:13:13.489 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:19.158+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:31:19.310+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:19.312086+0000)
, 2022-04-03T02:31:19.586+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:19.587542+0000)

2022-04-03T14:13:13.490 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:29.316+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:29.316774+0000)
, 2022-04-03T02:31:29.585+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:31:29.591+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:29.591710+0000)

2022-04-03T14:13:13.491 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:39.026+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:39.026571+0000)
, 2022-04-03T02:31:39.240+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:31:39.321+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:39.321725+0000)

2022-04-03T14:13:13.492 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:49.030+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:49.031811+0000)
, 2022-04-03T02:31:49.218+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:31:49.324+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:49.326098+0000)

2022-04-03T14:13:13.493 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:31:59.253+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:31:59.330+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:31:59.332066+0000)
, 2022-04-03T02:31:59.510+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.493 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:09.094 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:32:09.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T02:32:09.302+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.494 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:18.808+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:18.808532+0000)
, 2022-04-03T02:32:19.046+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:19.047344+0000)
, 2022-04-03T02:32:19.155+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.495 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:29.052+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:29.052505+0000)
, 2022-04-03T02:32:29.155+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:32:29.234+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.495 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:39.057+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:39.057739+0000)
, 2022-04-03T02:32:39.074+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:32:39.183 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.496 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:48.847+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:32:48.884+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:32:49.062+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:49.063151+0000)

2022-04-03T14:13:13.497 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:32:58.988+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:32:59.067+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:32:59.067959+0000)
, 2022-04-03T02:32:59.133+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.497 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:08.854+0000 7f7b589b7700 -1 osd.5 83 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:08.872+0000 7ff9b50d9700 -1 MDSIOContextBase: failed with -110, restarting...
,     -1> 2022-04-03T02:33:08.872+0000 7ff9b50d9700 -1 MDSIOContextBase: failed with -110, restarting...

2022-04-03T14:13:13.499 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:18.306+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:18.367+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:18.369039+0000)
, 2022-04-03T02:33:18.386+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.500 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:28.421+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:28.463+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:28.471+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.500 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:38.378+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:38.378528+0000)
, 2022-04-03T02:33:38.433+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:38.599+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.501 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:48.133+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:33:48.149+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:48.355+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.502 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:33:58.137+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:58.248+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:33:58.386+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:33:58.387767+0000)

2022-04-03T14:13:13.503 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:08.302+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:08.335+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:34:08.390+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:08.392257+0000)

2022-04-03T14:13:13.503 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:18.107+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:18.108869+0000)
, 2022-04-03T02:34:18.332+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:34:18.366+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.504 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:28.215+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:28.354+0000 7f7b589b7700 -1 osd.5 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:34:28.401+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:28.401736+0000)

2022-04-03T14:13:13.505 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:38.134+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:38.344+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:34:38.405+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:38.406497+0000)

2022-04-03T14:13:13.505 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:47.883+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:47.884667+0000)
, 2022-04-03T02:34:48.081+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:48.122+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:48.123933+0000)

2022-04-03T14:13:13.506 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:34:58.012+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:34:58.128+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:34:58.128895+0000)
, 2022-04-03T02:34:58.253+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.507 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:08.098+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:35:08.133+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:08.133702+0000)
, 2022-04-03T02:35:08.386+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.508 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:17.893+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:35:17.899+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:17.899655+0000)
, 2022-04-03T02:35:17.966+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.509 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:27.904+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:27.904700+0000)
, 2022-04-03T02:35:28.007+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:35:28.143+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:28.143855+0000)

2022-04-03T14:13:13.510 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:38.021+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:35:38.054+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:35:38.147+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:38.149112+0000)

2022-04-03T14:13:13.511 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:47.700+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:47.701201+0000)
, 2022-04-03T02:35:47.912+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:47.914077+0000)
, 2022-04-03T02:35:48.023+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.511 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:35:57.918+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:35:57.919415+0000)
, 2022-04-03T02:35:58.063+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:35:58.125+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.512 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:07.924+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:07.924790+0000)
, 2022-04-03T02:36:07.970+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:36:08.158+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.513 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:17.713+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:17.713642+0000)
, 2022-04-03T02:36:17.757+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:36:17.929+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:17.930050+0000)

2022-04-03T14:13:13.514 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:27.756+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:36:27.811+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:36:27.934+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:27.935072+0000)

2022-04-03T14:13:13.514 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:37.721+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:37.722610+0000)
, 2022-04-03T02:36:37.764+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:36:37.938+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:37.940121+0000)

2022-04-03T14:13:13.515 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:47.519+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:36:47.651+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:36:47.693+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.516 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:36:57.629+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:36:57.713+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:36:57.729+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:36:57.731152+0000)

2022-04-03T14:13:13.516 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:07.616+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:07.690+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:07.734+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:07.735412+0000)

2022-04-03T14:13:13.517 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:17.448+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:17.481+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:17.481319+0000)
, 2022-04-03T02:37:17.525+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.518 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:27.534+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:27.659+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:27.677+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.519 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:37.489+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:37.490730+0000)
, 2022-04-03T02:37:37.587+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:37:37.665+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.519 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:47.431+0000 7f7b589b7700 -1 osd.5 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:47.486+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:47.493+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:47.495240+0000)

2022-04-03T14:13:13.520 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:37:57.490+0000 7f7b589b7700 -1 osd.5 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:37:57.499+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:37:57.499949+0000)
, 2022-04-03T02:37:57.544+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.521 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:07.504+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:07.504510+0000)
, 2022-04-03T02:38:07.516+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:38:07.670+0000 7f7b589b7700 -1 osd.5 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.521 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:17.372+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:38:17.412+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:38:17.508+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:17.508367+0000)

2022-04-03T14:13:13.522 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:27.378+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:38:27.513+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:27.513194+0000)
, 2022-04-03T02:38:27.650+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.523 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:37.260+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:38:37.517+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:37.518450+0000)
, 2022-04-03T02:38:37.673+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.524 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:47.246+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:38:47.247630+0000)
, 2022-04-03T02:38:47.272+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:38:47.406+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.524 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:38:57.329+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:38:57.522+0000 7f7b589b7700 -1 osd.5 85 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:38:57.524+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.525 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:07.369+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:39:07.448+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:39:07.533+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:07.534191+0000)

2022-04-03T14:13:13.526 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:17.066+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:39:17.263+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:17.263386+0000)
, 2022-04-03T02:39:17.359+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.526 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:27.267+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:27.268866+0000)
, 2022-04-03T02:39:27.366+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:39:27.542+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:27.543349+0000)

2022-04-03T14:13:13.527 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:37.272+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:37.274000+0000)
, 2022-04-03T02:39:37.318+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:39:37.547+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:37.548252+0000)

2022-04-03T14:13:13.528 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:47.035+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:47.035918+0000)
, 2022-04-03T02:39:47.130+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:39:47.278+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:47.279140+0000)

2022-04-03T14:13:13.528 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:39:57.243+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:39:57.283+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:39:57.284425+0000)
, 2022-04-03T02:39:57.352+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.529 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:07.174+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:40:07.289+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:07.289708+0000)
, 2022-04-03T02:40:07.437+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.530 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:16.537+0000 7fa9c620e700 -1 osd.4 85 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:40:16.566+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:16.566900+0000)
, 2022-04-03T02:40:16.596+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.531 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:26.519+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:40:26.570+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:26.571481+0000)
, 2022-04-03T02:40:26.681+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.531 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:36.575+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:36.576501+0000)
, 2022-04-03T02:40:36.612+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:40:36.729+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.532 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:46.397+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:40:46.417+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:40:46.580+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:46.581343+0000)

2022-04-03T14:13:13.533 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:40:56.498+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:40:56.585+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:40:56.586312+0000)
, 2022-04-03T02:40:56.670+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.533 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:06.517+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:41:06.530+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:41:06.590+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:06.590943+0000)

2022-04-03T14:13:13.534 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:16.270+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:41:16.325+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:16.325801+0000)
, 2022-04-03T02:41:16.528+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.535 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:26.330+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:26.331084+0000)
, 2022-04-03T02:41:26.529+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:41:26.599+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:26.600356+0000)

2022-04-03T14:13:13.536 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:36.335+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:36.336761+0000)
, 2022-04-03T02:41:36.604+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:36.605383+0000)
, 2022-04-03T02:41:36.645+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.537 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:46.206+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:41:46.267+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:41:46.341+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:46.342134+0000)

2022-04-03T14:13:13.538 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:41:56.176+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:41:56.346+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:56.346983+0000)
, 2022-04-03T02:41:56.614+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:41:56.614713+0000)

2022-04-03T14:13:13.540 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:06.174+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:42:06.351+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:06.351845+0000)
, 2022-04-03T02:42:06.571+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.541 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:16.167+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:42:16.269+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T02:42:16.277+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.542 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:26.116+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:26.117634+0000)
, 2022-04-03T02:42:26.279+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:42:26.363+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:26.364920+0000)

2022-04-03T14:13:13.543 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:36.122+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:36.122901+0000)
, 2022-04-03T02:42:36.358+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:42:36.369+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:36.369852+0000)

2022-04-03T14:13:13.544 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:45.884+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:45.885385+0000)
, 2022-04-03T02:42:46.126+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:46.128080+0000)
, 2022-04-03T02:42:46.254+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.544 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:42:55.888+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:55.889463+0000)
, 2022-04-03T02:42:56.058+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:42:56.133+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:42:56.133595+0000)

2022-04-03T14:13:13.545 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:05.933+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:43:06.097+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:43:06.138+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:06.138429+0000)

2022-04-03T14:13:13.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:15.896+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:15.897275+0000)
, 2022-04-03T02:43:15.944+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:43:16.143+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:16.143942+0000)

2022-04-03T14:13:13.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:25.900+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:25.901389+0000)
, 2022-04-03T02:43:26.076+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:43:26.077+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.547 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:35.905+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:35.905754+0000)
, 2022-04-03T02:43:36.041+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:43:36.129+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.549 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:45.738+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:43:45.909+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:45.910488+0000)
, 2022-04-03T02:43:46.066+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.550 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:43:55.783+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:43:55.913+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:43:55.914839+0000)
, 2022-04-03T02:43:55.994+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:05.799+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:44:05.918+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:05.918979+0000)
, 2022-04-03T02:44:06.097+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:15.681+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:44:15.790+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:44:15.922+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:15.923651+0000)

2022-04-03T14:13:13.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:25.746+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:44:25.783+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:44:25.926+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:25.927949+0000)

2022-04-03T14:13:13.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:35.833+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:44:35.877+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:44:35.931+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:35.932579+0000)

2022-04-03T14:13:13.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:45.714+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:44:45.745+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:44:45.809+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.554 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:44:55.755+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:44:55.847+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:44:55.940+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:44:55.940763+0000)

2022-04-03T14:13:13.555 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:05.706+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:05.706398+0000)
, 2022-04-03T02:45:05.860+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:45:05.944+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:05.944835+0000)

2022-04-03T14:13:13.556 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:15.564+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:15.710+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:15.711531+0000)
, 2022-04-03T02:45:15.947+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:15.948706+0000)

2022-04-03T14:13:13.556 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:25.608+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:25.715+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:25.716696+0000)
, 2022-04-03T02:45:25.951+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:25.952664+0000)

2022-04-03T14:13:13.557 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:35.506+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:35.721+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:35.722142+0000)
, 2022-04-03T02:45:35.955+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:35.956617+0000)

2022-04-03T14:13:13.558 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:45.477+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:45.573+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:45.725+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:45.727241+0000)

2022-04-03T14:13:13.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:45:55.475+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:55.565+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:45:55.731+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:45:55.732237+0000)

2022-04-03T14:13:13.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:05.477+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:05.478025+0000)
, 2022-04-03T02:46:05.736+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:05.736931+0000)
, 2022-04-03T02:46:05.763+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.560 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:15.399+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:46:15.481+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:15.482903+0000)
, 2022-04-03T02:46:15.741+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:15.742353+0000)

2022-04-03T14:13:13.561 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:25.486+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:25.487812+0000)
, 2022-04-03T02:46:25.544+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:46:25.716+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.562 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:35.492+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:35.493326+0000)
, 2022-04-03T02:46:35.580+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:46:35.731+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.563 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:45.448+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:46:45.497+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:45.498475+0000)
, 2022-04-03T02:46:45.575+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.564 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:46:55.489+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:46:55.503+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:46:55.503803+0000)
, 2022-04-03T02:46:55.546+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:05.508+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:05.509129+0000)
, 2022-04-03T02:47:05.595+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:47:05.613+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:15.453+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:47:15.513+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:15.514428+0000)
, 2022-04-03T02:47:15.611+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.566 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:25.493+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:47:25.518+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:25.519674+0000)
, 2022-04-03T02:47:25.533+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.567 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:35.482+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:47:35.513+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:47:35.524+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:35.525061+0000)

2022-04-03T14:13:13.568 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:45.458+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:47:45.529+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:47:45.530229+0000)
, 2022-04-03T02:47:45.564+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.568 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:47:55.406+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:47:55.428+0000 7fa9c620e700 -1 osd.4 86 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:47:55.532+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.569 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:05.467+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:05.541+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:05.541150+0000)
, 2022-04-03T02:48:05.592+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.570 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:14.680+0000 7f7b589b7700 -1 osd.5 86 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:14.799+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:14.800396+0000)
, 2022-04-03T02:48:14.884+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.570 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:24.775+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:24.804+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:24.805402+0000)
, 2022-04-03T02:48:24.837+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.571 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:34.819+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:34.823+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:34.862+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:44.685+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:44.815+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:44.815504+0000)
, 2022-04-03T02:48:44.829+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.573 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:48:54.743+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:48:54.820+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:48:54.820547+0000)
, 2022-04-03T02:48:54.898+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.573 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:04.772+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:49:04.824+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:04.825445+0000)
, 2022-04-03T02:49:04.933+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:14.577+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:14.578131+0000)
, 2022-04-03T02:49:14.758+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:49:14.829+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:14.830642+0000)

2022-04-03T14:13:13.575 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:24.582+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:24.583150+0000)
, 2022-04-03T02:49:24.828+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:49:24.834+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:24.835482+0000)

2022-04-03T14:13:13.576 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:34.638+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:49:34.839+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:34.840243+0000)
, 2022-04-03T02:49:34.894+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.576 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:44.592+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:44.593170+0000)
, 2022-04-03T02:49:44.623+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:49:44.737+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.577 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:49:54.582+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:49:54.598+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:49:54.598673+0000)
, 2022-04-03T02:49:54.796+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:04.602+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:04.603566+0000)
, 2022-04-03T02:50:04.638+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:50:04.730+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:14.360+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:14.361415+0000)
, 2022-04-03T02:50:14.577+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:50:14.607+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:14.608985+0000)

2022-04-03T14:13:13.579 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:24.414+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:50:24.540+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:50:24.613+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:24.614162+0000)

2022-04-03T14:13:13.580 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:34.450+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:50:34.572+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:50:34.618+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:34.619464+0000)

2022-04-03T14:13:13.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:44.276+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:50:44.377+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:44.377423+0000)
, 2022-04-03T02:50:44.524+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:50:54.303+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:50:54.383+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:50:54.384088+0000)
, 2022-04-03T02:50:54.519+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 5 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.582 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:04.343+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:04.388+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:04.389417+0000)
, 2022-04-03T02:51:04.630+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:14.139+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:14.283+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:14.393+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:14.394635+0000)

2022-04-03T14:13:13.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:24.110+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:24.111547+0000)
, 2022-04-03T02:51:24.234+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:24.399+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:24.399733+0000)

2022-04-03T14:13:13.584 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:34.161+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:34.260+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:51:34.404+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:34.404867+0000)

2022-04-03T14:13:13.585 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:44.119+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:44.120267+0000)
, 2022-04-03T02:51:44.153+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:44.410+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:44.410281+0000)

2022-04-03T14:13:13.586 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:51:54.124+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:54.124625+0000)
, 2022-04-03T02:51:54.269+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:51:54.415+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:51:54.415574+0000)

2022-04-03T14:13:13.587 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:04.127+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:04.129024+0000)
, 2022-04-03T02:52:04.287+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:52:04.312+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.589 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:14.081+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:52:14.132+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:14.133581+0000)
, 2022-04-03T02:52:14.228+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.590 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:23.951+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:52:24.137+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:24.138402+0000)
, 2022-04-03T02:52:24.179+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:33.930+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:33.931887+0000)
, 2022-04-03T02:52:34.097+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:52:34.142+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:34.142604+0000)

2022-04-03T14:13:13.593 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:43.714+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:52:43.936+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:43.936746+0000)
, 2022-04-03T02:52:44.146+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:44.146527+0000)

2022-04-03T14:13:13.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:52:53.690+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:53.691108+0000)
, 2022-04-03T02:52:53.940+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:53.941566+0000)
, 2022-04-03T02:52:54.150+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:52:54.151125+0000)

2022-04-03T14:13:13.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:03.735+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:53:03.945+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:03.946774+0000)
, 2022-04-03T02:53:04.154+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:04.155407+0000)

2022-04-03T14:13:13.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:13.700+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:13.701730+0000)
, 2022-04-03T02:53:13.700+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:53:13.950+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:13.952248+0000)

2022-04-03T14:13:13.597 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:23.706+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:23.707115+0000)
, 2022-04-03T02:53:23.919+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:53:23.943+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.597 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:33.711+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:33.712235+0000)
, 2022-04-03T02:53:33.911+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:53:33.962+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:33.962896+0000)

2022-04-03T14:13:13.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:43.612+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:53:43.717+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:43.717355+0000)
, 2022-04-03T02:53:43.935+0000 7f7b589b7700 -1 osd.5 87 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:53:53.633+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:53:53.721+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:53:53.722470+0000)
, 2022-04-03T02:53:53.885+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.600 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:03.624+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:54:03.726+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:03.727863+0000)
, 2022-04-03T02:54:03.930+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.600 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:13.523+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:54:13.732+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:13.733477+0000)
, 2022-04-03T02:54:13.779+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:23.491+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:23.492140+0000)
, 2022-04-03T02:54:23.738+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:23.739001+0000)
, 2022-04-03T02:54:23.834+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:33.507+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:54:33.744+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:33.744444+0000)
, 2022-04-03T02:54:33.793+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:43.470+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:54:43.501+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:43.502122+0000)
, 2022-04-03T02:54:43.749+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:43.750159+0000)

2022-04-03T14:13:13.603 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:54:53.523+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:54:53.536+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:54:53.754+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:54:53.755259+0000)

2022-04-03T14:13:13.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:03.612+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:55:03.679+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:55:03.759+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:03.760320+0000)

2022-04-03T14:13:13.605 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:13.515+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:13.517228+0000)
, 2022-04-03T02:55:13.575+0000 7fa9c620e700 -1 osd.4 87 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:55:13.664+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.605 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:23.016+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:23.016774+0000)
, 2022-04-03T02:55:23.068+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:55:23.084+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:33.021+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:33.021899+0000)
, 2022-04-03T02:55:33.032+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:55:33.194+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:43.078+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:55:43.207+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T02:55:43.225+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:43.226065+0000)

2022-04-03T14:13:13.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:55:52.950+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:55:53.031+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:55:53.032398+0000)
, 2022-04-03T02:55:53.079+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:02.895+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:56:02.899+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:03.020+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:12.960+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:13.042+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:13.042739+0000)
, 2022-04-03T02:56:13.067+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:22.801+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:22.801639+0000)
, 2022-04-03T02:56:22.836+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:56:22.897+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:32.743+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:56:32.806+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:32.806757+0000)
, 2022-04-03T02:56:32.916+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:42.811+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:42.812079+0000)
, 2022-04-03T02:56:42.837+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:56:43.058+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:43.058288+0000)

2022-04-03T14:13:13.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:56:52.622+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:56:52.624+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:56:52.816+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:56:52.817318+0000)

2022-04-03T14:13:13.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:02.660+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:02.662+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:02.821+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:02.822522+0000)

2022-04-03T14:13:13.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:12.765+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:12.770+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:57:12.827+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:12.827724+0000)

2022-04-03T14:13:13.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:22.635+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:22.659+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:22.785+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:32.663+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:32.716+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:32.783+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:42.795+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:42.841+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:42.842896+0000)
, 2022-04-03T02:57:42.862+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:57:52.600+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:57:52.601661+0000)
, 2022-04-03T02:57:52.609+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:57:52.786+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:02.603+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:02.605+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:02.606894+0000)
, 2022-04-03T02:58:02.720+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:12.744+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:12.782+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:58:12.791+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:22.617+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:22.617350+0000)
, 2022-04-03T02:58:22.673+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:22.763+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:32.622+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:32.622616+0000)
, 2022-04-03T02:58:32.674+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:32.869+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:32.870034+0000)

2022-04-03T14:13:13.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:42.756+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:42.798+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:42.811+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:58:51.993+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:58:52.006+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:58:52.121+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:58:52.122584+0000)

2022-04-03T14:13:13.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:02.054+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:59:02.124+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:02.125978+0000)
, 2022-04-03T02:59:02.134+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:12.109+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:59:12.128+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:12.129310+0000)
, 2022-04-03T02:59:12.132+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:22.113+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:59:22.132+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:22.133037+0000)
, 2022-04-03T02:59:22.179+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.624 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:32.135+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:32.136065+0000)
, 2022-04-03T02:59:32.202+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:59:32.305+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.624 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:42.321+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:42.323039+0000)
, 2022-04-03T02:59:42.327+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T02:59:42.426+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.625 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T02:59:52.324+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T01:59:52.325853+0000)
, 2022-04-03T02:59:52.408+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T02:59:52.553+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:02.327+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:02.328572+0000)
, 2022-04-03T03:00:02.384+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:02.656+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:02.657874+0000)

2022-04-03T14:13:13.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:12.445+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:12.577+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:00:12.601+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.627 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:22.594+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:22.594+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:22.666+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:22.666987+0000)

2022-04-03T14:13:13.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:32.642+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:32.672+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:32.672688+0000)
, 2022-04-03T03:00:32.676+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:42.658+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:42.676+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:42.677920+0000)
, 2022-04-03T03:00:42.676+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.629 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:00:52.370+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:00:52.613+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:00:52.682+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:00:52.682987+0000)

2022-04-03T14:13:13.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:02.366+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:01:02.640+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:01:02.657+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:12.561+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:01:12.617+0000 7fa9c620e700 -1 osd.4 88 get_health_metrics reporting 39 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:01:12.618+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:22.361+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:22.361985+0000)
, 2022-04-03T03:01:22.428+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:01:22.513+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.632 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:32.365+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:32.366721+0000)
, 2022-04-03T03:01:32.460+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:01:32.503+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:42.370+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:42.371129+0000)
, 2022-04-03T03:01:42.476+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:01:42.519+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:01:52.223+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:01:52.303+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:01:52.374+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:01:52.375558+0000)

2022-04-03T14:13:13.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:02.174+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:02.204+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:02.204757+0000)
, 2022-04-03T03:02:02.266+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.635 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:12.208+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:12.210132+0000)
, 2022-04-03T03:02:12.219+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:02:12.382+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:12.383167+0000)

2022-04-03T14:13:13.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:21.998+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:02:22.163+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:02:22.170+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:32.031+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:02:32.121+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:02:32.219+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:32.220074+0000)

2022-04-03T14:13:13.637 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:42.123+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:02:42.191+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:02:42.224+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:42.225243+0000)

2022-04-03T14:13:13.638 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:02:51.974+0000 7f7b589b7700 -1 osd.5 88 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:02:51.991+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:02:51.992628+0000)
, 2022-04-03T03:02:52.211+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.638 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:01.967+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:03:01.997+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:01.997987+0000)
, 2022-04-03T03:03:02.190+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.639 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:12.003+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:12.003439+0000)
, 2022-04-03T03:03:12.127+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:03:12.206+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.640 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:21.226+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:03:21.244+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:21.245013+0000)
, 2022-04-03T03:03:21.417+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:21.417356+0000)

2022-04-03T14:13:13.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:31.285+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:03:31.331+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:03:31.421+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:31.422139+0000)

2022-04-03T14:13:13.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:41.278+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:03:41.426+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:41.426888+0000)
, 2022-04-03T03:03:41.476+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:03:51.124+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:03:51.259+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:03:51.260392+0000)
, 2022-04-03T03:03:51.397+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:01.264+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:01.265180+0000)
, 2022-04-03T03:04:01.276+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:04:01.333+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:13.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:11.345+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:04:11.431+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:04:11.440+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:11.440387+0000)

2022-04-03T14:13:13.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:21.041+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:21.041569+0000)
, 2022-04-03T03:04:21.275+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:21.275820+0000)
, 2022-04-03T03:04:21.413+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.645 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:31.279+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:31.281120+0000)
, 2022-04-03T03:04:31.334+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:04:31.391+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:41.285+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:41.286474+0000)
, 2022-04-03T03:04:41.393+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:04:41.452+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:41.453729+0000)

2022-04-03T14:13:13.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:04:50.949+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:04:51.057+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:51.057857+0000)
, 2022-04-03T03:04:51.291+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:04:51.291635+0000)

2022-04-03T14:13:13.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:01.062+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:01.063149+0000)
, 2022-04-03T03:05:01.296+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:01.297179+0000)
, 2022-04-03T03:05:01.336+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:13.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:11.151+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:05:11.278+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:05:11.302+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:11.302752+0000)

2022-04-03T14:13:13.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:20.839+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:05:21.073+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:21.074007+0000)
, 2022-04-03T03:05:21.197+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:31.049+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:05:31.058+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:05:31.078+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:31.079454+0000)

2022-04-03T14:13:13.650 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:41.084+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:41.085025+0000)
, 2022-04-03T03:05:41.122+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:05:41.127+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:05:50.810+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:05:50.841+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:05:50.842073+0000)
, 2022-04-03T03:05:51.080+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:00.846+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:00.847316+0000)
, 2022-04-03T03:06:01.095+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:01.095776+0000)
, 2022-04-03T03:06:01.215+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:10.889+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:06:11.100+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:11.101093+0000)
, 2022-04-03T03:06:11.187+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:20.666+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 7 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:06:20.856+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:20.857911+0000)
, 2022-04-03T03:06:20.907+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:30.862+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:30.863526+0000)
, 2022-04-03T03:06:30.967+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:06:31.110+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:31.111217+0000)

2022-04-03T14:13:13.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:40.867+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:40.869126+0000)
, 2022-04-03T03:06:40.875+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:06:41.115+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:41.116320+0000)

2022-04-03T14:13:13.655 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:06:50.594+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:06:50.733+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:06:50.874+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:06:50.874820+0000)

2022-04-03T14:13:13.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:00.761+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:00.879+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:00.880149+0000)
, 2022-04-03T03:07:01.092+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:10.869+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:10.885+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:10.885712+0000)
, 2022-04-03T03:07:11.094+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:20.534+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:20.633+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:20.859+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.658 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:30.767+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:30.895+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:30.896594+0000)
, 2022-04-03T03:07:30.912+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:40.848+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:40.901+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:40.902266+0000)
, 2022-04-03T03:07:40.931+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:07:50.541+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:07:50.541881+0000)
, 2022-04-03T03:07:50.757+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:07:50.860+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:00.727+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:08:00.884+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:08:00.908+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.661 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:10.702+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:08:10.814+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:08:10.858+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:20.398+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:20.399713+0000)
, 2022-04-03T03:08:20.555+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:20.556422+0000)
, 2022-04-03T03:08:20.772+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.663 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:30.560+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:30.561118+0000)
, 2022-04-03T03:08:30.813+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:08:30.854+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:40.564+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:40.565616+0000)
, 2022-04-03T03:08:40.766+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:08:40.794+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:08:50.185+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:50.186213+0000)
, 2022-04-03T03:08:50.413+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:50.414777+0000)
, 2022-04-03T03:08:50.569+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:08:50.570563+0000)

2022-04-03T14:13:13.665 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:00.419+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:00.420236+0000)
, 2022-04-03T03:09:00.574+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:00.575147+0000)
, 2022-04-03T03:09:00.714+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.666 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:10.426+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:10.426764+0000)
, 2022-04-03T03:09:10.579+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:10.580166+0000)
, 2022-04-03T03:09:10.691+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.668 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:20.200+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:20.201736+0000)
, 2022-04-03T03:09:20.356+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:09:20.430+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:20.432017+0000)

2022-04-03T14:13:13.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:30.206+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:30.207150+0000)
, 2022-04-03T03:09:30.436+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:30.437419+0000)
, 2022-04-03T03:09:30.588+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:30.589635+0000)

2022-04-03T14:13:13.670 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:40.334+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:09:40.442+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:40.442934+0000)
, 2022-04-03T03:09:40.593+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:40.594447+0000)

2022-04-03T14:13:13.671 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:09:50.175+0000 7fa9c620e700 -1 osd.4 89 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:09:50.217+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:09:50.218219+0000)
, 2022-04-03T03:09:50.280+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:00.129+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:10:00.222+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:00.223216+0000)
, 2022-04-03T03:10:00.452+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:00.453239+0000)

2022-04-03T14:13:13.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:10.111+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:10:10.228+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:10.228843+0000)
, 2022-04-03T03:10:10.458+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:10.458703+0000)

2022-04-03T14:13:13.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:19.973+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:10:19.993+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:19.994075+0000)
, 2022-04-03T03:10:20.106+0000 7f7b589b7700 -1 osd.5 89 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:29.620+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:10:29.718+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:10:29.718+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.676 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:39.621+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:39.622565+0000)
, 2022-04-03T03:10:39.630+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:10:39.779+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:49.478+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:49.478871+0000)
, 2022-04-03T03:10:49.560+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:10:49.625+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:49.627126+0000)

2022-04-03T14:13:13.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:10:59.581+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:10:59.631+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:10:59.631688+0000)
, 2022-04-03T03:10:59.760+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:09.489+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:09.489309+0000)
, 2022-04-03T03:11:09.635+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:09.636019+0000)
, 2022-04-03T03:11:09.714+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:19.266+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:19.267652+0000)
, 2022-04-03T03:11:19.411+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:11:19.492+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:19.494130+0000)

2022-04-03T14:13:13.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:29.490+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:11:29.498+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:29.499661+0000)
, 2022-04-03T03:11:29.627+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.682 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:39.504+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:39.504821+0000)
, 2022-04-03T03:11:39.616+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:11:39.641+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:49.283+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:49.283947+0000)
, 2022-04-03T03:11:49.294+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:11:49.509+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:49.510187+0000)

2022-04-03T14:13:13.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:11:59.288+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:59.289246+0000)
, 2022-04-03T03:11:59.504+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:11:59.514+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:11:59.514799+0000)

2022-04-03T14:13:13.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:09.293+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:09.294659+0000)
, 2022-04-03T03:12:09.518+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:09.519993+0000)
, 2022-04-03T03:12:09.663+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:09.664683+0000)

2022-04-03T14:13:13.685 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:19.057+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:19.058705+0000)
, 2022-04-03T03:12:19.195+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:12:19.298+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:19.299958+0000)

2022-04-03T14:13:13.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:29.131+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:12:29.305+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:29.305705+0000)
, 2022-04-03T03:12:29.426+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:39.071+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:12:39.310+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:39.311164+0000)
, 2022-04-03T03:12:39.393+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:48.940+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:12:49.008+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:12:49.075+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:49.075539+0000)

2022-04-03T14:13:13.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:12:59.007+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:12:59.080+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:12:59.080901+0000)
, 2022-04-03T03:12:59.224+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:09.085+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:09.086278+0000)
, 2022-04-03T03:13:09.100+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:13:09.325+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:09.326681+0000)

2022-04-03T14:13:13.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:18.775+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:13:19.044+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:19.090+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:19.091743+0000)

2022-04-03T14:13:13.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:28.949+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:29.074+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:13:29.096+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:29.097369+0000)

2022-04-03T14:13:13.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:39.046+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:39.059+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:13:39.101+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:39.102706+0000)

2022-04-03T14:13:13.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:48.711+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:48.711389+0000)
, 2022-04-03T03:13:48.856+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:13:49.094+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.692 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:13:58.842+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:13:58.891+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:13:59.113+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:13:59.113881+0000)

2022-04-03T14:13:13.693 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:08.732+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:14:08.850+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:14:09.118+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:09.119581+0000)

2022-04-03T14:13:13.694 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:18.719+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:14:18.725+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:18.726136+0000)
, 2022-04-03T03:14:18.858+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.695 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:28.760+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:14:28.903+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:14:29.012+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.695 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:38.807+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:14:38.844+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:14:39.130+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:48.601+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:48.602151+0000)
, 2022-04-03T03:14:48.720+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:14:48.740+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:48.740561+0000)

2022-04-03T14:13:13.697 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:14:58.705+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:14:58.744+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:14:58.745195+0000)
, 2022-04-03T03:14:58.749+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.697 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:08.732+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:08.748+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:08.749787+0000)
, 2022-04-03T03:15:08.841+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.698 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:18.617+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:18.618025+0000)
, 2022-04-03T03:15:18.666+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:18.753+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.699 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:28.668+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:28.712+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:28.759+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:28.759688+0000)

2022-04-03T14:13:13.700 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:38.649+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:38.665+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:38.764+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:38.764552+0000)

2022-04-03T14:13:13.700 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:48.505+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:48.570+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:48.598+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:15:58.582+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:15:58.638+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:15:58.639002+0000)
, 2022-04-03T03:15:58.728+0000 7f7b589b7700 -1 osd.5 90 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.702 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:08.643+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:08.644124+0000)
, 2022-04-03T03:16:08.685+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:16:08.777+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:08.778792+0000)

2022-04-03T14:13:13.702 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:18.428+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:18.430260+0000)
, 2022-04-03T03:16:18.607+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:16:18.647+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:18.649152+0000)

2022-04-03T14:13:13.703 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:28.599+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:16:28.653+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:28.654316+0000)
, 2022-04-03T03:16:28.775+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.708 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:38.590+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:16:38.641+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:16:38.659+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:38.659585+0000)

2022-04-03T14:13:13.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:48.335+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:16:48.448+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:48.448685+0000)
, 2022-04-03T03:16:48.499+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:16:58.452+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:58.453563+0000)
, 2022-04-03T03:16:58.478+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:16:58.669+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:16:58.670485+0000)

2022-04-03T14:13:13.710 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:08.408+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:17:08.457+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:08.458973+0000)
, 2022-04-03T03:17:08.674+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:08.675384+0000)

2022-04-03T14:13:13.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:18.223+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:18.224274+0000)
, 2022-04-03T03:17:18.238+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:17:18.376+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.712 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:28.299+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:17:28.396+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:17:28.435+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.712 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:38.292+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:17:38.315+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:17:38.344+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.713 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:48.075+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:17:48.188+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:17:48.240+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:48.240871+0000)

2022-04-03T14:13:13.714 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:17:58.161+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:17:58.188+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:17:58.245+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:17:58.246419+0000)

2022-04-03T14:13:13.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:08.191+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:18:08.251+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:08.252083+0000)
, 2022-04-03T03:18:08.285+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.715 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:18.018 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:18:18.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T03:18:18.118+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.716 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:27.481+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:18:27.500+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:27.501420+0000)
, 2022-04-03T03:18:27.619+0000 7fa9c620e700 -1 osd.4 90 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.717 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:37.393+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:18:37.506+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:37.506839+0000)
, 2022-04-03T03:18:37.602+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.717 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:47.421+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:18:47.512+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:18:47.512199+0000)
, 2022-04-03T03:18:47.691+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.718 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:18:57.397+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:18:57.413+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:18:57.448+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.719 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:07.354+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:07.414+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:19:07.433+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.720 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:17.306+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:17.390+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:17.477+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.720 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:27.293+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:27.294358+0000)
, 2022-04-03T03:19:27.323+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:27.507+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.721 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:37.299+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:37.299686+0000)
, 2022-04-03T03:19:37.380+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:37.447+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.722 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:47.282+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:47.305+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:47.305171+0000)
, 2022-04-03T03:19:47.441+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.722 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:19:57.099+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:57.254+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:19:57.309+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:19:57.310468+0000)

2022-04-03T14:13:13.723 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:07.117+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:07.315+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:07.315980+0000)
, 2022-04-03T03:20:07.405+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.724 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:17.140+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:17.205+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:20:17.320+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:17.321178+0000)

2022-04-03T14:13:13.725 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:27.099+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:20:27.133+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:27.326+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:27.326722+0000)

2022-04-03T14:13:13.725 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:37.132+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:20:37.141+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:37.285+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.4:6046 3.19 3:9d0ed013:::200.0000001b:head [write 495995~31387 [fadvise_dontneed] in=31387b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.726 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:47.149+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:47.171+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:20:47.336+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:47.337843+0000)

2022-04-03T14:13:13.727 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:20:56.918+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:20:56.919226+0000)
, 2022-04-03T03:20:57.113+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:20:57.241+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.728 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:06.964+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:21:07.005+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:07.246+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.728 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:16.994+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:17.055+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:21:17.069+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.729 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:26.933+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:26.933877+0000)
, 2022-04-03T03:21:26.971+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:21:26.994+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.730 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:36.938+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:36.939019+0000)
, 2022-04-03T03:21:36.981+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:21:37.045+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.730 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:46.930+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:46.942+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:46.943946+0000)
, 2022-04-03T03:21:47.039+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.731 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:21:56.861+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:21:56.904+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:21:56.948+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:21:56.948946+0000)

2022-04-03T14:13:13.732 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:06.883+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:22:06.952+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:06.953607+0000)
, 2022-04-03T03:22:06.996+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.733 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:16.958+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:16.958692+0000)
, 2022-04-03T03:22:17.003+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:22:17.028+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.733 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:26.963+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:26.963761+0000)
, 2022-04-03T03:22:26.975+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:22:27.119+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.734 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:36.936+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:22:36.969+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:36.969318+0000)
, 2022-04-03T03:22:37.066+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.735 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:46.906+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:22:46.956+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:22:46.973+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:46.974854+0000)

2022-04-03T14:13:13.736 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:22:56.830+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:22:56.858+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:22:56.858938+0000)
, 2022-04-03T03:22:56.881+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.736 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:06.742+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:06.774+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:23:06.863+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:06.864346+0000)

2022-04-03T14:13:13.737 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:16.746+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:16.828+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:23:16.849+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.738 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:26.666+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:26.666404+0000)
, 2022-04-03T03:23:26.716+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:26.815+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:36.621+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:36.671+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:36.671490+0000)
, 2022-04-03T03:23:36.759+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.739 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:46.651+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:46.676+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:23:46.677287+0000)
, 2022-04-03T03:23:46.694+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.740 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:23:56.514+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:56.601+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:23:56.621+0000 7f7b589b7700 -1 osd.5 91 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:06.504+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:24:06.650+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:24:06.687+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:06.688024+0000)

2022-04-03T14:13:13.742 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:16.515+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:24:16.693+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:16.693599+0000)
, 2022-04-03T03:24:16.733+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:26.463+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:26.463916+0000)
, 2022-04-03T03:24:26.502+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:24:26.604+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.744 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:36.450+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:24:36.468+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:36.469239+0000)
, 2022-04-03T03:24:36.639+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.744 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:46.473+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:46.474381+0000)
, 2022-04-03T03:24:46.510+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:24:46.623+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.745 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:24:56.478+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:24:56.479630+0000)
, 2022-04-03T03:24:56.494+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:24:56.573+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.746 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:06.348+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:25:06.484+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:06.485127+0000)
, 2022-04-03T03:25:06.634+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.747 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:16.490+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:16.490565+0000)
, 2022-04-03T03:25:16.556+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:25:16.567+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.747 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:26.373+0000 7fa9c620e700 -1 osd.4 91 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:25:26.495+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:26.496196+0000)
, 2022-04-03T03:25:26.552+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.748 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:35.549+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:25:35.561+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:25:35.735+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:35.737159+0000)

2022-04-03T14:13:13.749 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:45.679+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:25:45.741+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:25:45.742910+0000)
, 2022-04-03T03:25:45.921+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:25:55.568+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:25:55.650+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:25:55.739+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.750 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:05.710+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:26:05.752+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:05.754176+0000)
, 2022-04-03T03:26:05.761+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.751 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:15.759+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:15.759387+0000)
, 2022-04-03T03:26:15.803+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:26:15.830+0000 7f7b589b7700 -1 osd.5 92 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.752 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:25.528+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:25.528979+0000)
, 2022-04-03T03:26:25.574+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:26:25.764+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:25.764728+0000)

2022-04-03T14:13:13.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:35.533+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:35.534685+0000)
, 2022-04-03T03:26:35.769+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:35.770356+0000)
, 2022-04-03T03:26:35.772+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.753 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:45.704+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:26:45.774+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:45.775644+0000)
, 2022-04-03T03:26:45.799+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.754 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:26:55.519+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:26:55.544+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:26:55.545473+0000)
, 2022-04-03T03:26:55.723+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:05.553+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:27:05.591+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:27:05.656+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.755 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:15.598+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:27:15.682+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:27:15.792+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:15.792613+0000)

2022-04-03T14:13:13.756 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:25.560+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:25.560623+0000)
, 2022-04-03T03:27:25.596+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:27:25.639+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.757 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:35.565+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:35.566505+0000)
, 2022-04-03T03:27:35.576+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:27:35.613+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:45.570+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:45.571925+0000)
, 2022-04-03T03:27:45.596+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:27:45.678+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.758 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:27:55.380+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:27:55.486+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:27:55.576+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:27:55.577414+0000)

2022-04-03T14:13:13.759 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:05.297+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:28:05.447+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:05.487+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.760 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:15.364+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:28:15.439+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:15.553+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.761 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:25.152+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:25.282+0000 7f7b589b7700 -1 osd.5 92 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:25.340+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.761 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:35.141+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:35.142863+0000)
, 2022-04-03T03:28:35.222+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:35.345+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.762 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:45.173+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:45.360+0000 7f7b589b7700 -1 osd.5 92 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:28:45.486+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.763 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:28:55.055+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:55.056482+0000)
, 2022-04-03T03:28:55.151+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:28:55.152891+0000)
, 2022-04-03T03:28:55.230+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.764 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:05.061+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:05.061842+0000)
, 2022-04-03T03:29:05.157+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:05.157716+0000)
, 2022-04-03T03:29:05.295+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.765 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:15.162+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:15.162552+0000)
, 2022-04-03T03:29:15.336+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:29:15.391+0000 7f7b589b7700 -1 osd.5 92 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.766 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:24.863+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:24.864706+0000)
, 2022-04-03T03:29:25.071+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:25.072340+0000)
, 2022-04-03T03:29:25.166+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:25.167507+0000)

2022-04-03T14:13:13.767 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:34.925+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:29:35.076+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:35.077866+0000)
, 2022-04-03T03:29:35.171+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:35.172255+0000)

2022-04-03T14:13:13.768 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:45.082+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:45.083003+0000)
, 2022-04-03T03:29:45.176+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:45.176901+0000)
, 2022-04-03T03:29:45.375+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.768 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:29:54.879+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:54.880599+0000)
, 2022-04-03T03:29:54.887+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:29:55.087+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:29:55.088337+0000)

2022-04-03T14:13:13.769 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:04.833+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:30:04.886+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:04.886643+0000)
, 2022-04-03T03:30:05.093+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:05.093566+0000)

2022-04-03T14:13:13.770 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:14.891+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:14.892013+0000)
, 2022-04-03T03:30:15.098+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:15.099270+0000)
, 2022-04-03T03:30:15.191+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:15.191833+0000)

2022-04-03T14:13:13.771 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:24.659+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:24.660921+0000)
, 2022-04-03T03:30:24.779+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:30:24.896+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:24.897445+0000)

2022-04-03T14:13:13.772 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:34.758+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:30:34.805+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:30:34.902+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:34.903240+0000)

2022-04-03T14:13:13.773 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:44.908+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:44.908958+0000)
, 2022-04-03T03:30:44.934+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:30:45.115+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:30:45.116115+0000)

2022-04-03T14:13:13.774 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:30:54.803+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:30:54.841+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:30:54.853+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.776 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:04.699+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:31:04.738+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:31:04.889+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.777 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:14.799+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:31:14.924+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:14.925131+0000)
, 2022-04-03T03:31:15.005+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.778 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:24.721+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:31:24.878+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:31:24.922+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.780 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:34.757+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:31:34.853+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:31:34.935+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:34.935814+0000)

2022-04-03T14:13:13.781 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:44.935+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:31:44.940+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:44.941531+0000)
, 2022-04-03T03:31:44.970+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.782 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:31:54.709+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:54.709746+0000)
, 2022-04-03T03:31:54.807+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:31:54.946+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:31:54.947227+0000)

2022-04-03T14:13:13.783 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:04.715+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:04.715418+0000)
, 2022-04-03T03:32:04.801+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:32:04.953+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:04.953499+0000)

2022-04-03T14:13:13.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:14.767+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:32:14.959+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:14.959666+0000)
, 2022-04-03T03:32:15.001+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.784 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:24.194+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:32:24.217+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:32:24.248+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.785 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:34.171+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:34.173034+0000)
, 2022-04-03T03:32:34.231+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:32:34.255+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:34.256364+0000)

2022-04-03T14:13:13.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:44.175+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:44.176988+0000)
, 2022-04-03T03:32:44.258+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:44.259712+0000)
, 2022-04-03T03:32:44.288+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:32:54.173+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:32:54.179+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:54.180549+0000)
, 2022-04-03T03:32:54.261+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:32:54.262966+0000)

2022-04-03T14:13:13.787 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:04.170+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:33:04.183+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:04.184053+0000)
, 2022-04-03T03:33:04.265+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:04.266113+0000)

2022-04-03T14:13:13.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:14.197+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:33:14.269+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:14.269381+0000)
, 2022-04-03T03:33:14.484+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:24.271+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:24.272505+0000)
, 2022-04-03T03:33:24.322+0000 7fa9c620e700 -1 osd.4 92 get_health_metrics reporting 41 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:33:24.557+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.789 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:33.934+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:33:33.994+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:33.996244+0000)
, 2022-04-03T03:33:34.193+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:34.194832+0000)

2022-04-03T14:13:13.790 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:44.001+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:44.002410+0000)
, 2022-04-03T03:33:44.150+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:33:44.197+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:44.198801+0000)

2022-04-03T14:13:13.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:33:54.004+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:54.006198+0000)
, 2022-04-03T03:33:54.156+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:33:54.200+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:33:54.202203+0000)

2022-04-03T14:13:13.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:04.010+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:04.010920+0000)
, 2022-04-03T03:34:04.206+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:04.206602+0000)
, 2022-04-03T03:34:04.272+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.792 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:14.211+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:14.212148+0000)
, 2022-04-03T03:34:14.271+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:34:14.290+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:14.291383+0000)

2022-04-03T14:13:13.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:24.200+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:34:24.216+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:24.217383+0000)
, 2022-04-03T03:34:24.235+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.794 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:33.488+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:34:33.659+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:34:33.779+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:33.780535+0000)

2022-04-03T14:13:13.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:43.715+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:34:43.782+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:43.784198+0000)
, 2022-04-03T03:34:43.961+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:34:53.754+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:34:53.787+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:34:53.787873+0000)
, 2022-04-03T03:34:53.884+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.796 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:03.729+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:35:03.791+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:03.792013+0000)
, 2022-04-03T03:35:03.950+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:13.794+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:13.795690+0000)
, 2022-04-03T03:35:14.006+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:14.039+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:14.041238+0000)

2022-04-03T14:13:13.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:23.798+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:23.799498+0000)
, 2022-04-03T03:35:23.983+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:24.043+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:34.014+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:33.802+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:33.803463+0000)
, 2022-04-03T03:35:34.023+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:43.936+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:44.025+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:44.050+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:44.051785+0000)

2022-04-03T14:13:13.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:35:53.852+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:53.985+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:35:54.054+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:35:54.054623+0000)

2022-04-03T14:13:13.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:03.983+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:04.057+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:04.057769+0000)
, 2022-04-03T03:36:04.100+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:14.062+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:14.063344+0000)
, 2022-04-03T03:36:14.083+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:14.116+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:24.071+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:24.103+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:24.170+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.802 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:34.087+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:34.203+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:34.220+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.803 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:43.914+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:44.079+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:44.080426+0000)
, 2022-04-03T03:36:44.207+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:36:53.959+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:36:54.086+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:36:54.086360+0000)
, 2022-04-03T03:36:54.253+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:04.052+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:37:04.091+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:04.092033+0000)
, 2022-04-03T03:37:04.216+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.805 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:13.854+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:13.855715+0000)
, 2022-04-03T03:37:14.011+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:37:14.097+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:14.098034+0000)

2022-04-03T14:13:13.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:23.861+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:23.862026+0000)
, 2022-04-03T03:37:24.023+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:37:24.102+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:24.103714+0000)

2022-04-03T14:13:13.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:33.867+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:33.868234+0000)
, 2022-04-03T03:37:33.939+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:37:34.108+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:34.109561+0000)

2022-04-03T14:13:13.807 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:43.427+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:37:43.872+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:43.873744+0000)
, 2022-04-03T03:37:43.996+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:37:53.428+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:37:53.879+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:37:53.879376+0000)
, 2022-04-03T03:37:54.073+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:03.469+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:38:03.883+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:03.885082+0000)
, 2022-04-03T03:38:03.953+0000 7fa9c620e700 -1 osd.4 93 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:13.393+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:13.394665+0000)
, 2022-04-03T03:38:13.397+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:38:13.889+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:13.890803+0000)

2022-04-03T14:13:13.810 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:23.398+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:23.399681+0000)
, 2022-04-03T03:38:23.552+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:38:23.896+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:23.896994+0000)

2022-04-03T14:13:13.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:33.404+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:33.404818+0000)
, 2022-04-03T03:38:33.601+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:38:33.902+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:33.902704+0000)

2022-04-03T14:13:13.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:43.454+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:38:43.487+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:38:43.506+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.812 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:38:53.414+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:38:53.414698+0000)
, 2022-04-03T03:38:53.512+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:38:53.622+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:03.418+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:03.419495+0000)
, 2022-04-03T03:39:03.429+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:39:03.582+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:13.321+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:13.355+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:13.355926+0000)
, 2022-04-03T03:39:13.423+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:13.424713+0000)

2022-04-03T14:13:13.814 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:23.292+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:39:23.359+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:23.360731+0000)
, 2022-04-03T03:39:23.428+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:23.429763+0000)

2022-04-03T14:13:13.815 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:33.350+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:39:33.365+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:33.366445+0000)
, 2022-04-03T03:39:33.434+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:33.435063+0000)

2022-04-03T14:13:13.816 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:43.371+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:43.371850+0000)
, 2022-04-03T03:39:43.428+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:39:43.438+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:43.439822+0000)

2022-04-03T14:13:13.816 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:39:53.377+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:53.377317+0000)
, 2022-04-03T03:39:53.444+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:39:53.444390+0000)
, 2022-04-03T03:39:53.447+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:03.382+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:03.383258+0000)
, 2022-04-03T03:40:03.448+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:03.449234+0000)
, 2022-04-03T03:40:03.572+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.818 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:13.196+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:13.197854+0000)
, 2022-04-03T03:40:13.387+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:13.388721+0000)
, 2022-04-03T03:40:13.452+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:13.453722+0000)

2022-04-03T14:13:13.818 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:23.234+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:40:23.393+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:23.394177+0000)
, 2022-04-03T03:40:23.402+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.819 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:33.362+0000 7f7b589b7700 -1 osd.5 93 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:40:33.367+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:40:33.399+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:33.399925+0000)

2022-04-03T14:13:13.820 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:42.477+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:40:42.564+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:40:42.663+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.821 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:40:52.472+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:40:52.472938+0000)
, 2022-04-03T03:40:52.481+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:40:52.598+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.821 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:02.485+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:41:02.509+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:41:02.541+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.822 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:12.481+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:12.482080+0000)
, 2022-04-03T03:41:12.487+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:41:12.541+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:22.426+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:22.426929+0000)
, 2022-04-03T03:41:22.479+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:41:22.485+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:22.486933+0000)

2022-04-03T14:13:13.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:32.431+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:32.432288+0000)
, 2022-04-03T03:41:32.464+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:41:32.491+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:32.491772+0000)

2022-04-03T14:13:13.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:42.413+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:41:42.437+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:42.437940+0000)
, 2022-04-03T03:41:42.457+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:41:52.441+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:41:52.442186+0000)
, 2022-04-03T03:41:52.475+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:41:52.486+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:02.445+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:02.446824+0000)
, 2022-04-03T03:42:02.474+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:42:02.507+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:02.508285+0000)

2022-04-03T14:13:13.826 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:12.381+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:42:12.450+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:12.452210+0000)
, 2022-04-03T03:42:12.510+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.827 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:22.456+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:22.457381+0000)
, 2022-04-03T03:42:22.469+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:42:22.474+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:32.438+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:42:32.445+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:42:32.461+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:42:32.462068+0000)

2022-04-03T14:13:13.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:42.341+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:42:42.387+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:42:42.413+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.829 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:42:52.352+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:42:52.368+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:42:52.414+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:02.440+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:43:02.476+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:02.477424+0000)
, 2022-04-03T03:43:02.477+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:12.303+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:12.304113+0000)
, 2022-04-03T03:43:12.436+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:43:12.481+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:12.482814+0000)

2022-04-03T14:13:13.831 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:22.309+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:22.309967+0000)
, 2022-04-03T03:43:22.487+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:22.488216+0000)
, 2022-04-03T03:43:22.494+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.832 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:32.315+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:32.316014+0000)
, 2022-04-03T03:43:32.429+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:43:32.458+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.832 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:42.088+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:42.089062+0000)
, 2022-04-03T03:43:42.321+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:42.321813+0000)
, 2022-04-03T03:43:42.421+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:43:52.094+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:52.095006+0000)
, 2022-04-03T03:43:52.327+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:43:52.327880+0000)
, 2022-04-03T03:43:52.358+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.834 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:02.167+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:44:02.332+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:02.333720+0000)
, 2022-04-03T03:44:02.434+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:12.104+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:12.106077+0000)
, 2022-04-03T03:44:12.133+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:44:12.312+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.835 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:22.111+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:22.112208+0000)
, 2022-04-03T03:44:22.127+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:22.140+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.836 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:32.087+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:44:32.117+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:32.117726+0000)
, 2022-04-03T03:44:32.167+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:41.928+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:44:42.038+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:44:42.123+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:42.123895+0000)

2022-04-03T14:13:13.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:44:52.037+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:44:52.129+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:44:52.130080+0000)
, 2022-04-03T03:44:52.142+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.838 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:02.125+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:02.127+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:45:02.135+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:02.136052+0000)

2022-04-03T14:13:13.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:11.936+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:12.129+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:12.141+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:12.141990+0000)

2022-04-03T14:13:13.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:21.934+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:22.028+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:22.107+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:32.028+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:32.094+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:32.152+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:45:32.153374+0000)

2022-04-03T14:13:13.841 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:41.768+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:42.065+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:42.066+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:45:51.822+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:52.035+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:45:52.076+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:01.962+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:46:02.053+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:46:02.078+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.843 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:11.629+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:11.630739+0000)
, 2022-04-03T03:46:11.879+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:46:12.016+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:21.635+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:21.635855+0000)
, 2022-04-03T03:46:21.989+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:46:21.995+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:31.640+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:31.641347+0000)
, 2022-04-03T03:46:31.949+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:46:31.980+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:41.602+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:41.602743+0000)
, 2022-04-03T03:46:41.646+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:41.646521+0000)
, 2022-04-03T03:46:42.082+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:46:51.608+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:51.608550+0000)
, 2022-04-03T03:46:51.651+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:46:51.651770+0000)
, 2022-04-03T03:46:52.038+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:01.613+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:01.614406+0000)
, 2022-04-03T03:47:01.655+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:01.656786+0000)
, 2022-04-03T03:47:01.980+0000 7fa9c620e700 -1 osd.4 94 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:11.529+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:47:11.619+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:11.620259+0000)
, 2022-04-03T03:47:11.661+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:11.661807+0000)

2022-04-03T14:13:13.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:21.575+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:47:21.624+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:21.625993+0000)
, 2022-04-03T03:47:21.666+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:21.666650+0000)

2022-04-03T14:13:13.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:31.561+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:47:31.631+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:31.631517+0000)
, 2022-04-03T03:47:31.670+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:31.672045+0000)

2022-04-03T14:13:13.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:41.461+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:41.461908+0000)
, 2022-04-03T03:47:41.563+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:47:41.637+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:41.637533+0000)

2022-04-03T14:13:13.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:47:51.424+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:47:51.467+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:51.467447+0000)
, 2022-04-03T03:47:51.642+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:47:51.643247+0000)

2022-04-03T14:13:13.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:01.472+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:01.473227+0000)
, 2022-04-03T03:48:01.500+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:48:01.647+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:01.648718+0000)

2022-04-03T14:13:13.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:11.407+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:48:11.413+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:48:11.477+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:11.478647+0000)

2022-04-03T14:13:13.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:21.483+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:21.483703+0000)
, 2022-04-03T03:48:21.487+0000 7f7b589b7700 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:48:21.496+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.853 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:31.489+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:31.489562+0000)
, 2022-04-03T03:48:31.503+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T03:48:31.545+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.854 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:40.706+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:40.707026+0000)
, 2022-04-03T03:48:41.012+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:48:41.181+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:48:50.711+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:48:50.712268+0000)
, 2022-04-03T03:48:51.073+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:48:51.136+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:01.015+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:49:01.061+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:49:01.144+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.856 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:10.687+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:10.688450+0000)
, 2022-04-03T03:49:10.721+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:10.722242+0000)
, 2022-04-03T03:49:11.087+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:20.693+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:20.694276+0000)
, 2022-04-03T03:49:20.726+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:20.727538+0000)
, 2022-04-03T03:49:21.004+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:30.732+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:30.732711+0000)
, 2022-04-03T03:49:30.793+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:49:30.934+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:40.705+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:40.705703+0000)
, 2022-04-03T03:49:40.727+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:49:40.737+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:40.737892+0000)

2022-04-03T14:13:13.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:49:50.710+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:50.711864+0000)
, 2022-04-03T03:49:50.741+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:49:50.742202+0000)
, 2022-04-03T03:49:50.773+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:00.716+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:00.717535+0000)
, 2022-04-03T03:50:00.746+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:00.747155+0000)
, 2022-04-03T03:50:00.922+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:10.546+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:10.547524+0000)
, 2022-04-03T03:50:10.622+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:50:10.722+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:10.722929+0000)

2022-04-03T14:13:13.861 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:20.552+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:20.553623+0000)
, 2022-04-03T03:50:20.727+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:20.728547+0000)
, 2022-04-03T03:50:20.755+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:20.756763+0000)

2022-04-03T14:13:13.862 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:30.734+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:30.734664+0000)
, 2022-04-03T03:50:30.761+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:30.761914+0000)
, 2022-04-03T03:50:30.775+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:40.364+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:50:40.564+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:40.564742+0000)
, 2022-04-03T03:50:40.644+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:50:50.337+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:50.338125+0000)
, 2022-04-03T03:50:50.569+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:50.571099+0000)
, 2022-04-03T03:50:50.745+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:50:50.746845+0000)

2022-04-03T14:13:13.864 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:00.575+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:00.576837+0000)
, 2022-04-03T03:51:00.643+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:00.751+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:00.752529+0000)

2022-04-03T14:13:13.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:10.253+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:10.349+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:10.350386+0000)
, 2022-04-03T03:51:10.582+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:10.582917+0000)

2022-04-03T14:13:13.866 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:20.355+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:20.355863+0000)
, 2022-04-03T03:51:20.394+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:20.588+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:20.589169+0000)

2022-04-03T14:13:13.867 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:30.361+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:30.361879+0000)
, 2022-04-03T03:51:30.594+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:30.594938+0000)
, 2022-04-03T03:51:30.747+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 15 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.867 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:40.184+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:40.275+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:40.367+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:40.368105+0000)

2022-04-03T14:13:13.868 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:51:50.131+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:50.284+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:51:50.372+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:51:50.373903+0000)

2022-04-03T14:13:13.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:00.289+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:00.378+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:00.379563+0000)
, 2022-04-03T03:52:00.612+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:00.613004+0000)

2022-04-03T14:13:13.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:10.128+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:10.207+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:10.219+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:20.029+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:20.136+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:20.389+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:20.390495+0000)

2022-04-03T14:13:13.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:30.211+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:30.395+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:52:30.395801+0000)
, 2022-04-03T03:52:30.428+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.872 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:40.088+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:40.129+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:52:40.178+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.872 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:52:50.089+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:52:50.151+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:52:50.192+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.873 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:00.097+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:53:00.098+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:00.330+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:09.997+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:10.070+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:10.083+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:20.130+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:20.133+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:53:20.140+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:30.092+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:30.203+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:30.254+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.876 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:39.854+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:39.854760+0000)
, 2022-04-03T03:53:39.995+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:40.159+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:49.858+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:53:49.859742+0000)
, 2022-04-03T03:53:49.976+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:50.169+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:53:59.985+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:53:59.998+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:54:00.091+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:09.859+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:09.859684+0000)
, 2022-04-03T03:54:09.869+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:09.870263+0000)
, 2022-04-03T03:54:10.013+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:19.865+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:19.865703+0000)
, 2022-04-03T03:54:19.875+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:19.875480+0000)
, 2022-04-03T03:54:19.969+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:29.880+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:29.880364+0000)
, 2022-04-03T03:54:30.009+0000 7fa9c620e700 -1 osd.4 95 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:54:30.057+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:39.705+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:39.706261+0000)
, 2022-04-03T03:54:39.875+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:39.876745+0000)
, 2022-04-03T03:54:39.884+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:39.885454+0000)

2022-04-03T14:13:13.881 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:49.710+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:49.712089+0000)
, 2022-04-03T03:54:49.881+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:49.882559+0000)
, 2022-04-03T03:54:49.889+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:49.890489+0000)

2022-04-03T14:13:13.881 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:54:59.864+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:54:59.887+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:59.888356+0000)
, 2022-04-03T03:54:59.894+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:54:59.895753+0000)

2022-04-03T14:13:13.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:09.723+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:09.723869+0000)
, 2022-04-03T03:55:09.782+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:55:09.893+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:09.894025+0000)

2022-04-03T14:13:13.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:19.729+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:19.729317+0000)
, 2022-04-03T03:55:19.773+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:55:19.818+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.884 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:29.888+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:55:29.905+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:29.905774+0000)
, 2022-04-03T03:55:29.910+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:29.910312+0000)

2022-04-03T14:13:13.884 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:39.740+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:39.741339+0000)
, 2022-04-03T03:55:39.754+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:55:39.900+0000 7f7b589b7700 -1 osd.5 95 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:48.988+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:55:49.264+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:55:49.276+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.886 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:55:59.166+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:55:59.249+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:55:59.517+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:55:59.518562+0000)

2022-04-03T14:13:13.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:09.098+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:09.220+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:56:09.431+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:18.953+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:19.086+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:19.096+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:29.192+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:29.192+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:29.411+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.889 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:39.129+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:39.186+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:39.282+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.889 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:48.949+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:48.950337+0000)
, 2022-04-03T03:56:48.949+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:48.950550+0000)
, 2022-04-03T03:56:49.130+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.890 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:56:58.955+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:56:58.956180+0000)
, 2022-04-03T03:56:59.211+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:56:59.387+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.891 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:08.960+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:08.961192+0000)
, 2022-04-03T03:57:09.121+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:57:09.350+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:18.844+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:57:18.965+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:18.966174+0000)
, 2022-04-03T03:57:18.966+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:18.966355+0000)

2022-04-03T14:13:13.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:28.971+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:28.971412+0000)
, 2022-04-03T03:57:28.971+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:28.971854+0000)
, 2022-04-03T03:57:29.128+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.893 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:38.975+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:38.976484+0000)
, 2022-04-03T03:57:38.975+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:38.976815+0000)
, 2022-04-03T03:57:39.126+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.894 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:48.819+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:48.820655+0000)
, 2022-04-03T03:57:48.981+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:48.981903+0000)
, 2022-04-03T03:57:48.981+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:48.981960+0000)

2022-04-03T14:13:13.895 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:57:58.952+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:57:58.986+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:58.987434+0000)
, 2022-04-03T03:57:58.986+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:57:58.987727+0000)

2022-04-03T14:13:13.895 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:08.991+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:08.992602+0000)
, 2022-04-03T03:58:08.991+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:08.992893+0000)
, 2022-04-03T03:58:09.117+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.896 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:18.255+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:58:18.439+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:58:18.504+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:28.502+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:58:28.505+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:58:28.605+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:28.605632+0000)

2022-04-03T14:13:13.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:38.498+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:58:38.520+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:58:38.608+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:38.609525+0000)

2022-04-03T14:13:13.898 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:48.588+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:58:48.588+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:58:48.612+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:48.613772+0000)

2022-04-03T14:13:13.899 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:58:58.616+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:58:58.618044+0000)
, 2022-04-03T03:58:58.661+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:58:58.781+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.900 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:08.621+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:08.622349+0000)
, 2022-04-03T03:59:08.680+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:59:08.744+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.901 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:18.626+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:18.627051+0000)
, 2022-04-03T03:59:18.762+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:59:18.785+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.902 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:28.781+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:59:28.862+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T03:59:28.865+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:28.866933+0000)

2022-04-03T14:13:13.903 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:38.803+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:59:38.869+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:38.870469+0000)
, 2022-04-03T03:59:38.899+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.904 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:48.779+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:59:48.847+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T03:59:48.874+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:48.875590+0000)

2022-04-03T14:13:13.906 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T03:59:58.852+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T03:59:58.880+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:58.881253+0000)
, 2022-04-03T03:59:59.031+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T02:59:59.032372+0000)

2022-04-03T14:13:13.907 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:08.885+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:08.885984+0000)
, 2022-04-03T04:00:08.934+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:00:08.985+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.908 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:18.048+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:18.048369+0000)
, 2022-04-03T04:00:18.084+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:00:18.208+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:28.150+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:00:28.271+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:00:28.661+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:28.663257+0000)

2022-04-03T14:13:13.910 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:38.079+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:00:38.279+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:00:38.666+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:38.667344+0000)

2022-04-03T14:13:13.910 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:48.089+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:00:48.267+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:00:48.670+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:48.671422+0000)

2022-04-03T14:13:13.911 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:00:58.383+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:00:58.584+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:00:58.674+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:00:58.675303+0000)

2022-04-03T14:13:13.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:08.424+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:01:08.510+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:01:08.679+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:08.679637+0000)

2022-04-03T14:13:13.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:18.433+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:01:18.522+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:01:18.610+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.913 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:28.465+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:01:28.686+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:28.688046+0000)
, 2022-04-03T04:01:28.694+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:38.510+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:01:38.647+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:01:38.690+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:38.692184+0000)

2022-04-03T14:13:13.915 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:48.551+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:01:48.565+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:01:48.695+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:48.696390+0000)

2022-04-03T14:13:13.915 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:01:58.547+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:01:58.701+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:01:58.702405+0000)
, 2022-04-03T04:01:58.743+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.916 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:08.574+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:08.619+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:02:08.708+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:08.708339+0000)

2022-04-03T14:13:13.917 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:18.593+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:18.612+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:18.714+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:18.714284+0000)

2022-04-03T14:13:13.918 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:28.403+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:28.565+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:28.677+0000 7fa9c620e700 -1 osd.4 96 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.918 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:38.505+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:38.589+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:38.712+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:48.369+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:48.593+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:48.656+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.920 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:02:58.131+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:02:58.132408+0000)
, 2022-04-03T04:02:58.420+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:02:58.586+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.920 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:08.181+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:08.359+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:08.659+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:18.280+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:18.420+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:18.613+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.922 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:28.246+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:03:28.329+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:28.473+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:38.217+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:03:38.346+0000 7f7b589b7700 -1 osd.5 96 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:38.481+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:47.689+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:03:47.768+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:47.768809+0000)
, 2022-04-03T04:03:47.997+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:47.998122+0000)

2022-04-03T14:13:13.924 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:03:57.615+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:57.692+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:03:57.773+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:03:57.775021+0000)

2022-04-03T14:13:13.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:07.704+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:07.714+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:04:07.780+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:07.781081+0000)

2022-04-03T14:13:13.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:17.728+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:17.732+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:17.786+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:17.786875+0000)

2022-04-03T14:13:13.926 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:27.645+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:04:27.784+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:27.791+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.927 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:37.643+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:04:37.674+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:37.798+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:37.798837+0000)

2022-04-03T14:13:13.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:47.714+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:47.743+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:04:47.803+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:04:47.804807+0000)

2022-04-03T14:13:13.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:04:57.355+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:57.639+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:04:57.657+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.929 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:07.398+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:05:07.540+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:05:07.770+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.930 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:17.410+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:05:17.561+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:05:17.823+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:05:17.823469+0000)

2022-04-03T14:13:13.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:27.234+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:05:27.486+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:05:27.548+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:37.308+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:05:37.535+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:05:37.633+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.932 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:47.422+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:05:47.475+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:05:47.793+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.933 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:05:57.373+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:05:57.421+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:05:57.540+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.934 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:07.380+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:07.495+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:06:07.599+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.934 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:17.507+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:17.561+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:06:17.654+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.935 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:27.264+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:27.542+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:27.547+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.936 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:37.269+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:37.486+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:06:37.548+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.937 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:47.278+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:47.521+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:06:47.577+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.938 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:06:57.268+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:06:57.269412+0000)
, 2022-04-03T04:06:57.387+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:06:57.522+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.939 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:07.274+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:07.274779+0000)
, 2022-04-03T04:07:07.462+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:07:07.590+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.941 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:17.280+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:17.280594+0000)
, 2022-04-03T04:07:17.488+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:07:17.553+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.942 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:27.256+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:27.257409+0000)
, 2022-04-03T04:07:27.285+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:27.286404+0000)
, 2022-04-03T04:07:27.508+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.943 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:37.261+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:37.262726+0000)
, 2022-04-03T04:07:37.291+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:37.292543+0000)
, 2022-04-03T04:07:37.371+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.944 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:47.247+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:07:47.266+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:47.267342+0000)
, 2022-04-03T04:07:47.297+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:47.298104+0000)

2022-04-03T14:13:13.946 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:07:57.257+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:07:57.270+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:07:57.271581+0000)
, 2022-04-03T04:07:57.272+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.947 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:07.275+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:07.276074+0000)
, 2022-04-03T04:08:07.276+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:08:07.276+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.947 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:17.260+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:08:17.260+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:08:17.280+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:17.281224+0000)

2022-04-03T14:13:13.948 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:27.129+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:08:27.171+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:27.172719+0000)
, 2022-04-03T04:08:27.184+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.949 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:37.177+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:37.178676+0000)
, 2022-04-03T04:08:37.197+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:08:37.290+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:37.291515+0000)

2022-04-03T14:13:13.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:47.143+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:08:47.183+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:47.184882+0000)
, 2022-04-03T04:08:47.295+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:47.296487+0000)

2022-04-03T14:13:13.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:08:57.028+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:08:57.041+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:08:57.190+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:08:57.191010+0000)

2022-04-03T14:13:13.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:07.055+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:09:07.071+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:09:07.196+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:07.196907+0000)

2022-04-03T14:13:13.952 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:16.969+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:16.969790+0000)
, 2022-04-03T04:09:17.202+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:17.202529+0000)
, 2022-04-03T04:09:17.310+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.952 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:26.975+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:26.976072+0000)
, 2022-04-03T04:09:27.050+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:09:27.207+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:27.208494+0000)

2022-04-03T14:13:13.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:37.060+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:09:37.065+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:09:37.213+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:37.213963+0000)

2022-04-03T14:13:13.954 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:46.987+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:46.988398+0000)
, 2022-04-03T04:09:47.039+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:09:47.218+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:47.220274+0000)

2022-04-03T14:13:13.955 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:09:56.994+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:09:56.994651+0000)
, 2022-04-03T04:09:57.005+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:09:57.163+0000 7fa9c620e700 -1 osd.4 97 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.955 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:07.000+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:07.000908+0000)
, 2022-04-03T04:10:07.078+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:10:07.163+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.956 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:17.006+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:17.006783+0000)
, 2022-04-03T04:10:17.054+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:10:17.171+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:26.802+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:10:27.011+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:27.013044+0000)
, 2022-04-03T04:10:27.098+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:36.663+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:10:36.913+0000 7f7b589b7700 -1 osd.5 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:10:37.018+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:37.019573+0000)

2022-04-03T14:13:13.958 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:46.255+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:46.256327+0000)
, 2022-04-03T04:10:46.293+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:10:46.356+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:46.358200+0000)

2022-04-03T14:13:13.959 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:10:56.012+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:10:56.030+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:56.030738+0000)
, 2022-04-03T04:10:56.261+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:10:56.261941+0000)

2022-04-03T14:13:13.960 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:06.036+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:06.036835+0000)
, 2022-04-03T04:11:06.050+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:11:06.232+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.960 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:16.042+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:16.042814+0000)
, 2022-04-03T04:11:16.274+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:16.274224+0000)
, 2022-04-03T04:11:16.315+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.961 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:25.975+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:11:26.038+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:11:26.047+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:26.048920+0000)

2022-04-03T14:13:13.962 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:35.999+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:11:36.054+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:36.055114+0000)
, 2022-04-03T04:11:36.094+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.963 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:46.059+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:46.061100+0000)
, 2022-04-03T04:11:46.107+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:11:46.277+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:13.963 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:11:55.828+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:11:55.960+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:11:56.067+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:11:56.067602+0000)

2022-04-03T14:13:13.964 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:05.804+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:05.957+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:06.073+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:06.074196+0000)

2022-04-03T14:13:13.965 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:15.992+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:16.087+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:16.088024+0000)
, 2022-04-03T04:12:16.311+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:16.311312+0000)

2022-04-03T14:13:13.966 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:25.625+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:25.974+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:26.001+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.967 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:35.608+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:36.055+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:36.081+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.967 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:46.105+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:46.106115+0000)
, 2022-04-03T04:12:46.109+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:46.200+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.968 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:12:55.474+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:12:55.475132+0000)
, 2022-04-03T04:12:55.595+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:12:56.052+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.969 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:05.480+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:05.481159+0000)
, 2022-04-03T04:13:05.534+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:13:06.117+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:06.117620+0000)

2022-04-03T14:13:13.970 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:15.580+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:13:16.123+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:16.123496+0000)
, 2022-04-03T04:13:16.128+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.970 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:25.492+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:25.493391+0000)
, 2022-04-03T04:13:25.505+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:13:25.544+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.971 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:35.497+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:35.498663+0000)
, 2022-04-03T04:13:35.520+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:13:35.539+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.972 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:45.478+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:13:45.503+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:45.503804+0000)
, 2022-04-03T04:13:46.140+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:46.141422+0000)

2022-04-03T14:13:13.973 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:13:55.444+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:13:55.455+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:55.455770+0000)
, 2022-04-03T04:13:55.509+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:13:55.509818+0000)

2022-04-03T14:13:13.973 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:05.454+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:14:05.460+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:05.461097+0000)
, 2022-04-03T04:14:05.515+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:05.515708+0000)

2022-04-03T14:13:13.974 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:15.466+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:15.466809+0000)
, 2022-04-03T04:14:15.521+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:15.521652+0000)
, 2022-04-03T04:14:15.533+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.975 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:25.393+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:14:25.403+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:14:25.471+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:25.472169+0000)

2022-04-03T14:13:13.975 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:35.418+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:14:35.476+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:35.477658+0000)
, 2022-04-03T04:14:35.519+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.976 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:45.496+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:14:45.538+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:45.539365+0000)
, 2022-04-03T04:14:45.561+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.977 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:14:55.407+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:55.407485+0000)
, 2022-04-03T04:14:55.486+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:55.488275+0000)
, 2022-04-03T04:14:55.544+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:14:55.545138+0000)

2022-04-03T14:13:13.977 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:05.493+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:05.493722+0000)
, 2022-04-03T04:15:05.504+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:15:05.551+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:05.551339+0000)

2022-04-03T14:13:13.978 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:15.498+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:15.498983+0000)
, 2022-04-03T04:15:15.556+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:15.557202+0000)
, 2022-04-03T04:15:15.556+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.979 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:25.425+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:25.426369+0000)
, 2022-04-03T04:15:25.466+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:15:25.503+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:25.504436+0000)

2022-04-03T14:13:13.980 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:35.431+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:35.432794+0000)
, 2022-04-03T04:15:35.457+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:15:35.506+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.980 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:45.438+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:45.439258+0000)
, 2022-04-03T04:15:45.479+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:15:45.487+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.981 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:15:55.219+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:55.219708+0000)
, 2022-04-03T04:15:55.401+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:15:55.444+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:15:55.444733+0000)

2022-04-03T14:13:13.982 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:05.225+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:05.225950+0000)
, 2022-04-03T04:16:05.353+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:16:05.441+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.982 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:15.313+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:16:15.382+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:16:15.455+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:15.457042+0000)

2022-04-03T14:13:13.983 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:25.088+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:16:25.219+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:16:25.237+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:25.238688+0000)

2022-04-03T14:13:13.984 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:35.085+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:16:35.244+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:35.245059+0000)
, 2022-04-03T04:16:35.273+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.984 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:45.145+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:16:45.224+0000 7f7b589b7700 -1 osd.5 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:16:45.250+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:16:45.251328+0000)

2022-04-03T14:13:13.985 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:16:54.913+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:16:55.094+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:16:55.115+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.986 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:04.862+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:05.039+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:05.112+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.987 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:15.110+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:15.207+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:17:15.268+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:15.269793+0000)

2022-04-03T14:13:13.987 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:24.687+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:24.963+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:25.175+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.988 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:34.681+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:17:34.809+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:35.090+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.989 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:44.960+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:45.152+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:17:45.205+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.989 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:17:54.588+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:54.588456+0000)
, 2022-04-03T04:17:54.652+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:17:54.652641+0000)
, 2022-04-03T04:17:54.788+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.990 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:04.593+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:04.593917+0000)
, 2022-04-03T04:18:04.658+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:04.658556+0000)
, 2022-04-03T04:18:04.826+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.991 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:14.662+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:14.664153+0000)
, 2022-04-03T04:18:14.700+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:18:15.141+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.992 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:24.593+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:18:24.602+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:24.603804+0000)
, 2022-04-03T04:18:24.669+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:24.670059+0000)

2022-04-03T14:13:13.992 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:34.608+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:34.609284+0000)
, 2022-04-03T04:18:34.675+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:18:34.676276+0000)
, 2022-04-03T04:18:34.733+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.993 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:44.719+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:18:44.757+0000 7fa9c620e700 -1 osd.4 98 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:18:45.020+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.994 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:18:54.221+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:18:54.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T04:18:54.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.995 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:04.108+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:04.278+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:19:04.314+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.995 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:14.291+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:14.342+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:14.343480+0000)
, 2022-04-03T04:19:14.431+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:13.996 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:24.338+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:19:24.349+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:24.350333+0000)
, 2022-04-03T04:19:24.364 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:13.997 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:34.121+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:34.286+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:34.336+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:13.998 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:44.229+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:44.271+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:44.362+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:44.362654+0000)

2022-04-03T14:13:13.998 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:19:54.276+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:54.292+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:19:54.368+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:19:54.368735+0000)

2022-04-03T14:13:13.999 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:03.837+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:04.132+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:04.180+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.000 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:14.113+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:14.283+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:14.297+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.001 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:24.197+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:24.356+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:24.386+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:24.387584+0000)

2022-04-03T14:13:14.001 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:33.742+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:33.743388+0000)
, 2022-04-03T04:20:33.986+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:34.102+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.002 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:43.928+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:43.996+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:44.305+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.003 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:20:53.890+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:53.917+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:20:54.404+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:20:54.405153+0000)

2022-04-03T14:13:14.003 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:03.716+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:21:03.760+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:03.761094+0000)
, 2022-04-03T04:21:03.842+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.004 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:13.766+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:13.767621+0000)
, 2022-04-03T04:21:13.832+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:21:13.931+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.005 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:23.860+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:21:23.877+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:21:23.989+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.005 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:33.702+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:33.703503+0000)
, 2022-04-03T04:21:33.743+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:21:33.779+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:33.779651+0000)

2022-04-03T14:13:14.006 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:43.762+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:21:43.784+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:43.785255+0000)
, 2022-04-03T04:21:43.981+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.007 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:21:53.791+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:21:53.791692+0000)
, 2022-04-03T04:21:53.846+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:21:54.026+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.008 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:03.674+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:03.675356+0000)
, 2022-04-03T04:22:03.718+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:03.719394+0000)
, 2022-04-03T04:22:03.796+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:03.797286+0000)

2022-04-03T14:13:14.008 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:13.723+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:13.724833+0000)
, 2022-04-03T04:22:13.802+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:13.803904+0000)
, 2022-04-03T04:22:13.893+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.009 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:23.762+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:22:23.808+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:23.809934+0000)
, 2022-04-03T04:22:23.857+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.010 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:33.734+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:33.735453+0000)
, 2022-04-03T04:22:33.737+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:22:33.815+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:33.816016+0000)

2022-04-03T14:13:14.010 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:43.740+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:43.740536+0000)
, 2022-04-03T04:22:43.821+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:43.821884+0000)
, 2022-04-03T04:22:43.967+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.011 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:22:53.745+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:53.746018+0000)
, 2022-04-03T04:22:53.827+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:22:53.827547+0000)
, 2022-04-03T04:22:54.003+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.012 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:03.653+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:23:03.711+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:03.713013+0000)
, 2022-04-03T04:23:03.750+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:03.751752+0000)

2022-04-03T14:13:14.012 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:13.734+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:23:13.756+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:13.756921+0000)
, 2022-04-03T04:23:13.838+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:13.839133+0000)

2022-04-03T14:13:14.013 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:23.761+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:23.762406+0000)
, 2022-04-03T04:23:23.844+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:23.844724+0000)
, 2022-04-03T04:23:23.902+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.014 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:33.730+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:33.730980+0000)
, 2022-04-03T04:23:33.748+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:23:33.767+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:33.767898+0000)

2022-04-03T14:13:14.015 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:43.772+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:43.773189+0000)
, 2022-04-03T04:23:43.772+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:23:43.815+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.015 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:23:53.742+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:53.743123+0000)
, 2022-04-03T04:23:53.778+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:53.778476+0000)
, 2022-04-03T04:23:53.862+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:23:53.862705+0000)

2022-04-03T14:13:14.016 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:03.523+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:03.524080+0000)
, 2022-04-03T04:24:03.642+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:24:03.748+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:03.748975+0000)

2022-04-03T14:13:14.017 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:13.685+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:24:13.745+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:24:13.754+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:13.754952+0000)

2022-04-03T14:13:14.017 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:23.563+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:24:23.760+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:23.761131+0000)
, 2022-04-03T04:24:23.761+0000 7fa9c620e700 -1 osd.4 99 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.018 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:33.507+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:24:33.542+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:33.543254+0000)
, 2022-04-03T04:24:33.591+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.019 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:43.549+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:43.549739+0000)
, 2022-04-03T04:24:43.580+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:24:43.773+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:43.773530+0000)

2022-04-03T14:13:14.020 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:24:53.613+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:24:53.682+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:24:53.779+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:24:53.780058+0000)

2022-04-03T14:13:14.020 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:03.235+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:25:03.560+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:03.561459+0000)
, 2022-04-03T04:25:03.603+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.021 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:13.485+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:13.503+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:25:13.523+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.022 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:23.368+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:23.556+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:25:23.572+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:23.573547+0000)

2022-04-03T14:13:14.022 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:33.030+0000 7f7b589b7700 -1 osd.5 99 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:25:33.212+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:25:33.463+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.023 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:43.215+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:25:43.517+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:25:43.585+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:43.586030+0000)

2022-04-03T14:13:14.024 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:25:52.679+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:25:52.770+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:25:52.815+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:25:52.816222+0000)

2022-04-03T14:13:14.025 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:02.587+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:26:02.597+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:02.598321+0000)
, 2022-04-03T04:26:02.745+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.025 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:12.606+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:12.607468+0000)
, 2022-04-03T04:26:12.687+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:26:12.817+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.026 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:22.612+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:22.613275+0000)
, 2022-04-03T04:26:22.624+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:26:22.759+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.027 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:32.361+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:26:32.421+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:26:32.619+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:32.620149+0000)

2022-04-03T14:13:14.027 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:42.597+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:26:42.601+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:26:42.626+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:42.626303+0000)

2022-04-03T14:13:14.028 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:26:52.493+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:26:52.631+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:26:52.633011+0000)
, 2022-04-03T04:26:52.668+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.029 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:02.360+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:27:02.381+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:27:02.614+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.030 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:12.501+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:27:12.599+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:27:12.645+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:12.645750+0000)

2022-04-03T14:13:14.030 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:22.509+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:27:22.650+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:22.651728+0000)
, 2022-04-03T04:27:22.653+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.031 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:31.989+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:27:31.989542+0000)
, 2022-04-03T04:27:32.450+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:27:32.451+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.032 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:42.280+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:27:42.481+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:27:42.576+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.032 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:27:52.310+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:27:52.525+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:27:52.545+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.033 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:01.934+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:02.006+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:02.007474+0000)
, 2022-04-03T04:28:02.335+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.034 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:12.133+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:12.501+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:12.613+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:22.218+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:22.384+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:22.564+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:31.928+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:31.928778+0000)
, 2022-04-03T04:28:32.025+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:32.025616+0000)
, 2022-04-03T04:28:32.221+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.036 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:42.031+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:42.031926+0000)
, 2022-04-03T04:28:42.051+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:42.478+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.037 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:28:52.037+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:28:52.038273+0000)
, 2022-04-03T04:28:52.141+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:28:52.428+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.038 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:01.944+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:01.945341+0000)
, 2022-04-03T04:29:01.996+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:02.043+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:02.044635+0000)

2022-04-03T14:13:14.038 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:11.970+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:12.049+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:12.050399+0000)
, 2022-04-03T04:29:12.140+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.039 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:21.980+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:21.995+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:22.055+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:22.056003+0000)

2022-04-03T14:13:14.040 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:31.905+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:31.949+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:31.950091+0000)
, 2022-04-03T04:29:31.951+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.040 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:41.835+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:41.956+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:41.956436+0000)
, 2022-04-03T04:29:41.966+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:41.967069+0000)

2022-04-03T14:13:14.041 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:29:51.746+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:29:51.961+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:51.962838+0000)
, 2022-04-03T04:29:51.971+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:29:51.972235+0000)

2022-04-03T14:13:14.042 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:01.643+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:30:01.721+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:01.750+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:01.751672+0000)

2022-04-03T14:13:14.043 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:11.667+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:11.756+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:11.757909+0000)
, 2022-04-03T04:30:11.974+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:11.975508+0000)

2022-04-03T14:13:14.043 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:21.623+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:21.763+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:21.764172+0000)
, 2022-04-03T04:30:21.981+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:21.981996+0000)

2022-04-03T14:13:14.044 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:31.642+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:31.645+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:30:31.660+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:41.538+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:41.750+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:41.776+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:41.776919+0000)

2022-04-03T14:13:14.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:30:51.539+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:30:51.771+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:30:51.782+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:30:51.783338+0000)

2022-04-03T14:13:14.046 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:01.468+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:31:01.535+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:31:01.646+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:14.047 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:11.583+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:31:11.584+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:31:11.795+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:11.795841+0000)

2022-04-03T14:13:14.048 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:21.585+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:31:21.695+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:31:21.801+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:21.802123+0000)

2022-04-03T14:13:14.048 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:31.439+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:31:31.648+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
, 2022-04-03T04:31:31.690+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.049 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:41.475+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:31:41.786+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:31:41.813+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:31:41.814414+0000)

2022-04-03T14:13:14.050 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:31:51.578+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:31:51.597+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:31:51.751+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.050 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:01.422+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:01.523+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:32:01.557+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:11.394+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:11.567+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:32:11.580+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.052 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:21.567+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:21.573+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:32:21.634+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:31.177+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:31.177815+0000)
, 2022-04-03T04:32:31.643+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:32:31.651+0000 7f5d8484e700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied

2022-04-03T14:13:14.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:41.182+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:41.184026+0000)
, 2022-04-03T04:32:41.628+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:41.661+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.054 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:32:51.188+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:32:51.189779+0000)
, 2022-04-03T04:32:51.606+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:32:51.702+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.055 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:01.113+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:33:01.194+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:01.195776+0000)
, 2022-04-03T04:33:01.546+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:11.128+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:33:11.201+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:11.201804+0000)
, 2022-04-03T04:33:11.553+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:21.165+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:33:21.207+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:21.207500+0000)
, 2022-04-03T04:33:21.673+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:31.100+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:31.100765+0000)
, 2022-04-03T04:33:31.213+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:31.213467+0000)
, 2022-04-03T04:33:31.311+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:41.105+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:41.107053+0000)
, 2022-04-03T04:33:41.218+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:33:41.219789+0000)
, 2022-04-03T04:33:41.418+0000 7fa9c620e700 -1 osd.4 100 get_health_metrics reporting 45 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:33:50.736+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:33:50.779+0000 7f7b589b7700 -1 osd.5 100 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:33:50.858+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:00.710+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:34:00.809+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:00.890+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:10.757+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:34:10.828+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:10.865+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:20.827+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:34:20.839+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:20.912+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:34:20.912591+0000)

2022-04-03T14:13:14.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:30.491+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:34:30.821+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:34:30.842+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:40.531+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:34:40.657+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:34:40.766+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:34:50.504+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:34:50.579+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:34:50.606+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:00.266+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:00.267284+0000)
, 2022-04-03T04:35:00.446+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:35:00.551+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:10.272+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:10.273384+0000)
, 2022-04-03T04:35:10.517+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:35:10.591+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:20.278+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:20.278818+0000)
, 2022-04-03T04:35:20.536+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:35:20.672+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.066 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:30.214+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:35:30.284+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:30.284870+0000)
, 2022-04-03T04:35:30.535+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 24 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:40.244+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:35:40.289+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:40.290900+0000)
, 2022-04-03T04:35:40.628+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.068 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:35:50.295+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:35:50.297166+0000)
, 2022-04-03T04:35:50.326+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:35:50.551+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.068 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:00.196+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:00.197372+0000)
, 2022-04-03T04:36:00.302+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:00.303175+0000)
, 2022-04-03T04:36:00.419+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:10.202+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:10.203247+0000)
, 2022-04-03T04:36:10.280+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:36:10.308+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:10.309268+0000)

2022-04-03T14:13:14.070 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:20.209+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:20.209677+0000)
, 2022-04-03T04:36:20.314+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:20.315170+0000)
, 2022-04-03T04:36:20.412+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.071 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:30.193+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:30.194053+0000)
, 2022-04-03T04:36:30.215+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:30.216007+0000)
, 2022-04-03T04:36:30.313+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.071 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:40.198+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:40.199512+0000)
, 2022-04-03T04:36:40.221+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:40.222381+0000)
, 2022-04-03T04:36:40.273+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:36:50.228+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:50.229005+0000)
, 2022-04-03T04:36:50.254+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:36:50.332+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:36:50.333100+0000)

2022-04-03T14:13:14.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:00.206+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:37:00.209+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:00.210108+0000)
, 2022-04-03T04:37:00.233+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:00.235134+0000)

2022-04-03T14:13:14.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:10.215+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:10.215800+0000)
, 2022-04-03T04:37:10.240+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:10.241405+0000)
, 2022-04-03T04:37:10.269+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:20.220+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:20.220878+0000)
, 2022-04-03T04:37:20.247+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:20.247718+0000)
, 2022-04-03T04:37:20.261+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:30.124+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:37:30.134+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:37:30.224+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:30.225960+0000)

2022-04-03T14:13:14.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:40.070+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:37:40.230+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:40.231590+0000)
, 2022-04-03T04:37:40.259+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:40.260138+0000)

2022-04-03T14:13:14.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:50.044+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:50.045380+0000)
, 2022-04-03T04:37:50.236+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:50.237182+0000)
, 2022-04-03T04:37:50.265+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:37:50.266584+0000)

2022-04-03T14:13:14.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:37:59.952+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:38:00.050+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:00.051495+0000)
, 2022-04-03T04:38:00.242+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:00.242805+0000)

2022-04-03T14:13:14.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:09.924+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:38:10.057+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:10.057887+0000)
, 2022-04-03T04:38:10.247+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:10.248212+0000)

2022-04-03T14:13:14.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:20.064+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:20.064434+0000)
, 2022-04-03T04:38:20.105+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:38:20.253+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:20.253903+0000)

2022-04-03T14:13:14.079 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:29.832+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:38:30.069+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:30.071130+0000)
, 2022-04-03T04:38:30.226+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:39.878+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:38:40.076+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:40.077428+0000)
, 2022-04-03T04:38:40.235+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:49.937+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:38:50.082+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:38:50.083950+0000)
, 2022-04-03T04:38:50.184+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:38:59.815+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:38:59.930+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:00.089+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:00.090456+0000)

2022-04-03T14:13:14.082 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:09.663+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:10.054+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:10.096+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:10.097165+0000)

2022-04-03T14:13:14.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:19.684+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:20.034+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:20.073+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.085 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:29.427+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:29.428369+0000)
, 2022-04-03T04:39:29.660+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:29.777+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.086 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:39.454+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:39.551+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:39.655+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:49.505+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:49.535+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:49.654+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:39:59.444+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:39:59.445400+0000)
, 2022-04-03T04:39:59.572+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:39:59.594+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:09.451+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:09.451381+0000)
, 2022-04-03T04:40:09.519+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:40:09.561+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.091 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:19.362+0000 7fa9c620e700 -1 osd.4 101 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:40:19.457+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:19.457320+0000)
, 2022-04-03T04:40:19.634+0000 7f7b589b7700 -1 osd.5 101 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.093 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:29.322+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:29.324083+0000)
, 2022-04-03T04:40:29.366+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:29.367587+0000)
, 2022-04-03T04:40:29.461+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:29.462955+0000)

2022-04-03T14:13:14.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:39.329+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:39.330192+0000)
, 2022-04-03T04:40:39.373+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:39.373869+0000)
, 2022-04-03T04:40:39.467+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:39.468725+0000)

2022-04-03T14:13:14.095 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:49.357+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:40:49.379+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:49.380210+0000)
, 2022-04-03T04:40:49.474+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:40:49.474674+0000)

2022-04-03T14:13:14.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:40:58.843+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:40:58.900 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf
, 2022-04-03T04:40:58.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:14.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:08.688+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:41:08.751+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:41:08.753+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:18.649+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:41:18.680+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:18.770+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:28.649+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:28.809+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:41:28.973 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf

2022-04-03T14:13:14.102 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:38.503+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:41:38.504455+0000)
, 2022-04-03T04:41:38.615+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:41:38.677+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:48.540+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:41:48.672+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:41:48.703+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.104 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:41:58.589+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:41:58.758+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:41:58.898+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.106 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:08.217+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:08.217741+0000)
, 2022-04-03T04:42:08.256+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:42:08.377+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:08.378172+0000)

2022-04-03T14:13:14.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:18.220+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:18.222247+0000)
, 2022-04-03T04:42:18.323+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:42:18.368+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.108 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:28.251+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:42:28.384+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:28.386011+0000)
, 2022-04-03T04:42:28.389+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:38.282+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:42:38.388+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:38.390251+0000)
, 2022-04-03T04:42:38.439+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:38.440534+0000)

2022-04-03T14:13:14.111 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:48.325+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:42:48.392+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:48.394032+0000)
, 2022-04-03T04:42:48.444+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:48.444830+0000)

2022-04-03T14:13:14.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:42:58.413+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:42:58.441+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:42:58.447+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:42:58.449008+0000)

2022-04-03T14:13:14.113 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:08.350+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:43:08.401+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:08.401908+0000)
, 2022-04-03T04:43:08.453+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:08.453454+0000)

2022-04-03T14:13:14.115 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:18.311+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:43:18.404+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:18.405686+0000)
, 2022-04-03T04:43:18.456+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:18.457467+0000)

2022-04-03T14:13:14.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:28.460+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:28.461578+0000)
, 2022-04-03T04:43:28.500+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:43:28.528+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.117 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:38.437+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:43:38.466+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:38.466706+0000)
, 2022-04-03T04:43:38.560+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:38.560733+0000)

2022-04-03T14:13:14.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:48.495+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:43:48.559+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:43:48.564+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:48.566054+0000)

2022-04-03T14:13:14.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:43:58.572+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:43:58.572444+0000)
, 2022-04-03T04:43:58.584+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:43:58.662+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:08.436+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:08.467+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:44:08.485+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:08.486123+0000)

2022-04-03T14:13:14.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:18.491+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:18.492438+0000)
, 2022-04-03T04:44:18.498+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:18.504+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:28.497+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:28.498699+0000)
, 2022-04-03T04:44:28.590+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:28.591415+0000)
, 2022-04-03T04:44:28.590+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)

2022-04-03T14:13:14.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:38.383+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:38.447+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:38.447725+0000)
, 2022-04-03T04:44:38.496+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:48.453+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:48.453529+0000)
, 2022-04-03T04:44:48.503+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:44:48.510+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:48.510529+0000)

2022-04-03T14:13:14.128 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:44:58.492+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:44:58.516+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:58.516481+0000)
, 2022-04-03T04:44:58.609+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:44:58.609945+0000)

2022-04-03T14:13:14.129 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:08.316+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:08.316778+0000)
, 2022-04-03T04:45:08.465+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:08.465158+0000)
, 2022-04-03T04:45:08.478+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.130 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:18.326+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:45:18.432+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:45:18.470+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:18.471048+0000)

2022-04-03T14:13:14.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:28.475+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:28.476763+0000)
, 2022-04-03T04:45:28.489+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:45:28.535+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:28.536538+0000)

2022-04-03T14:13:14.133 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:38.335+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:38.336315+0000)
, 2022-04-03T04:45:38.379+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:45:38.387+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:48.249+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:45:48.342+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:48.343023+0000)
, 2022-04-03T04:45:48.487+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:48.488003+0000)

2022-04-03T14:13:14.136 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:45:58.349+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:58.349512+0000)
, 2022-04-03T04:45:58.424+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:45:58.493+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:45:58.493743+0000)

2022-04-03T14:13:14.137 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:08.118+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:08.247+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:08.355+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:08.355529+0000)

2022-04-03T14:13:14.138 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:18.250+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:18.261+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:18.319+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)

2022-04-03T14:13:14.140 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:28.243+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:28.258+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:46:28.367+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:46:28.368302+0000)

2022-04-03T14:13:14.141 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:38.190+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:46:38.252+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:38.253+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.142 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:48.190+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:46:48.197+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:48.341+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.144 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:46:58.253+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:46:58.284+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:46:58.333+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.144 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:07.767+0000 7fe76394c700 -1 osd.1 38 get_health_metrics reporting 38 slow ops, oldest is osd_repop(mds.2.11:6010 3.a e38/18 3:50d9c3ab:::202.0000000f:head v 38'218, mlcod=38'215)
, 2022-04-03T04:47:08.097+0000 7f7b589b7700 -1 osd.5 102 get_health_metrics reporting 3 slow ops, oldest is osd_op(mds.1.9:12282 3.17 3:ea85266a:::201.00000000:head [writefull 0~90 [fadvise_dontneed] in=90b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:47:08.190+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.145 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:17.693+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:17.694894+0000)
, 2022-04-03T04:47:18.102+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:47:18.140+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.146 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:28.159+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:47:28.220+0000 7f84c6f6c700 -1 osd.2 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.649:4 3.5 3.fcd6c765 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e44)
, 2022-04-03T04:47:28.225+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.147 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:37.615+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:37.616373+0000)
, 2022-04-03T04:47:37.706+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:37.706748+0000)
, 2022-04-03T04:47:38.156+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.147 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:47.622+0000 7fe6af90b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:47.622796+0000)
, 2022-04-03T04:47:47.712+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:47.712847+0000)
, 2022-04-03T04:47:48.207+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.148 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:47:57.718+0000 7f84bad46700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:47:57.718923+0000)
, 2022-04-03T04:47:58.163+0000 7ff8bff51700 -1 osd.3 38 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.2.11:6010 3.a 3:50d9c3ab:::202.0000000f:head [write 3090660~1995 [fadvise_dontneed] in=1995b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:47:58.389+0000 7fa9c620e700 -1 osd.4 102 get_health_metrics reporting 46 slow ops, oldest is osd_op(mds.0.4:6025 3.19 3:9d0ed013:::200.0000001b:head [write 345646~35505 [fadvise_dontneed] in=35505b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)

2022-04-03T14:13:14.149 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-03T04:48:07.431+0000 7ff8b3d2b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:07.432626+0000)
, 2022-04-03T04:48:07.473+0000 7fe6bbb31700 -1 osd.0 38 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.1.9:12278 3.3 3:c75ebc8c:::10000003eb4.02c00000:head [omap-set-header in=274b,omap-set-vals in=1867b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e38)
, 2022-04-03T04:48:07.563+0000 7fe757726700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T03:48:07.564586+0000)

2022-04-03T14:13:29.479 INFO:scrape:Found 13 distinct failure reasons
2022-04-03T14:13:29.481 INFO:scrape:Failure: Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid 252550c8-b2c9-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.5 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'
2022-04-03T14:13:29.481 INFO:scrape:42 jobs
2022-04-03T14:13:29.482 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{frag', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.483 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/5', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/misc}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsstress}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/iozone}', 'workunit/suites/pjd}', 'wsync/{no}}', 'wsync/{yes}}']
2022-04-03T14:13:29.483 INFO:scrape:
2022-04-03T14:13:29.484 INFO:scrape:Dead: b'2022-04-03T06:11:09.294 DEBUG:teuthology.exit:Finished running handlers'
2022-04-03T14:13:29.485 INFO:scrape:ceph version 17.0.0-11430-gcc9ed509 (cc9ed509562d767c1a5d8fc115593bc1453dfea1) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f4f0e6d3c20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b8) [0x55de2b9dd19e]
 5: (KernelDevice::_aio_thread()+0x1275) [0x55de2c62fae5]
 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55de2c63d041]
 7: /lib64/libpthread.so.0(+0x817f) [0x7f4f0e6c917f]
 8: clone()
2022-04-03T14:13:29.485 INFO:scrape:11 jobs: ['6774069', '6774011', '6773944', '6774191', '6774024', '6773988', '6773984', '6774111', '6774132', '6773931', '6774167']
2022-04-03T14:13:29.486 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.486 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{whitelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.487 INFO:scrape:
2022-04-03T14:13:29.487 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on smithi006 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh'
2022-04-03T14:13:29.488 INFO:scrape:9 jobs: ['6774176', '6773889', '6773868', '6774202', '6773958', '6773916', '6774148', '6773903', '6774056']
2022-04-03T14:13:29.489 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/3', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.489 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.490 INFO:scrape:
2022-04-03T14:13:29.490 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2022-04-03T14:13:29.490 INFO:scrape:4 jobs: ['6774140', '6774172', '6773878', '6774198']
2022-04-03T14:13:29.491 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.492 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.492 INFO:scrape:
2022-04-03T14:13:29.493 INFO:scrape:Dead: b'2022-04-03T03:26:49.336 DEBUG:teuthology.exit:Finished running handlers'
2022-04-03T14:13:29.493 INFO:scrape:14 jobs: ['6773879', '6774161', '6774122', '6773983', '6774079', '6774212', '6773970', '6774036', '6774091', '6773949', '6773999', '6774080', '6774163', '6774037']
2022-04-03T14:13:29.494 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.494 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/fs/snaps}}', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{pg-warn', 'overrides/{whitelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'roles', 'scrub/no', 'session_timeout', 'standby-replay', 'syntax', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}']
2022-04-03T14:13:29.495 INFO:scrape:
2022-04-03T14:13:29.495 INFO:scrape:Failure: 
2022-04-03T14:13:29.496 INFO:scrape:1 jobs: ['6774196']
2022-04-03T14:13:29.497 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.497 INFO:scrape:
2022-04-03T14:13:29.498 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-04-03T14:13:29.498 INFO:scrape:2 jobs: ['6774090', '6774020']
2022-04-03T14:13:29.499 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.499 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.500 INFO:scrape:
2022-04-03T14:13:29.500 INFO:scrape:Failure: "2022-04-03T00:06:44.887013+0000 osd.1 (osd.1) 14 : cluster [ERR] 5.19s0 deep-scrub : stat mismatch, got 0/26 objects, 0/0 clones, 0/26 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/34679 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-04-03T14:13:29.501 INFO:scrape:1 jobs: ['6774073']
2022-04-03T14:13:29.501 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.502 INFO:scrape:
2022-04-03T14:13:29.502 INFO:scrape:Failure: Command failed on smithi017 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)"
2022-04-03T14:13:29.503 INFO:scrape:2 jobs: ['6773959', '6773851']
2022-04-03T14:13:29.503 INFO:scrape:suites intersection: []
2022-04-03T14:13:29.504 INFO:scrape:suites union: ['1-ceph', '1-mount', '2-logrotate}', '2-workload/dbench', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/single-container-host}', 'distro/{ubuntu_latest}', 'fs/cephadm/multivolume/{0-start', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.504 INFO:scrape:
2022-04-03T14:13:29.505 INFO:scrape:Failure: Test failure: test_perf_stats_stale_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2022-04-03T14:13:29.505 INFO:scrape:1 jobs: ['6774071']
2022-04-03T14:13:29.506 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/metrics}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.506 INFO:scrape:
2022-04-03T14:13:29.507 INFO:scrape:Failure: "2022-04-03T01:02:17.231716+0000 mon.a (mon.0) 479 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-04-03T14:13:29.507 INFO:scrape:2 jobs: ['6774134', '6773846']
2022-04-03T14:13:29.508 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'osd}', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.508 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.509 INFO:scrape:
2022-04-03T14:13:29.509 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2022-04-03T14:13:29.510 INFO:scrape:1 jobs: ['6774193']
2022-04-03T14:13:29.510 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/acls}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.511 INFO:scrape:
2022-04-03T14:13:29.511 INFO:scrape:Failure: Command failed on smithi165 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cd52e30a-b2d8-11ec-8c36-001a4aab830c -- ceph mon dump -f json'
2022-04-03T14:13:29.512 INFO:scrape:1 jobs: ['6774009']
2022-04-03T14:13:29.512 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-04-03T14:13:29.512 INFO:scrape:
2022-04-03T14:13:29.533 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/
failed:  63
dead:    17
running: 11
waiting: 0
queued:  0
passed:  282


Fail
=================================================================
[6773845]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
-----------------------------------------------------------------
time:   00:12:37
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773845/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=40ff553b6f054fc7904cd548653b183a

    Command failed on smithi173 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    af7f1046-b2c0-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.173 --skip-admin-label
    && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773846]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:49:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773846/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773846/

    "2022-04-02T20:20:22.797233+0000 mon.a (mon.0) 615 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6773851]  fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host}
-----------------------------------------------------------------
time:   00:21:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773851/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773851/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77f19e39a7b54bee8d61a0bfa5b7eefa

    Command failed on smithi190 with status 1: "(cd /home/ubuntu/cephtest &&
    exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1
    sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
    daemon-helper kill ceph-fuse -f --admin-socket
    '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1
    --client_fs=bar)"

[6773867]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:48
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773867/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773867/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60bf356c9bea4d4b9accb44df761577f

    Command failed on smithi022 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    c6a77402-b2c5-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.22 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773868]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:24:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773868/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773868/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0a8c9ded188240b7b95e7c4b3f90f903

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi059 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6773876]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773876/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773876/
sentry: https://sentry.ceph.com/organizations/ceph/?query=be7bd2108a3040d9883094264d4a2f56

    Command failed on smithi005 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    252550c8-b2c9-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.5 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773881]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773881/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773881/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3b530ec0b43e4a06a0cc109937ccd75e

    Command failed on smithi105 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    c4d2e798-b2c9-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.105 --skip-admin-label
    && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773889]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:25:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773889/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773889/
sentry: https://sentry.ceph.com/organizations/ceph/?query=39e3b0d258fb4577865cd4b0204d2270

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi142 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6773890]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773890/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773890/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fda35b3d6211433897f8188112254ced

    Command failed on smithi061 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    468ea622-b2cb-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.61 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773897]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773897/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773897/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4261db4e280144c49eb30e048477b2bb

    Command failed on smithi027 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    1d7e26f8-b2cc-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773903]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:23:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773903/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773903/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ed0605b60fec47d898a2b24add59402b

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi006 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6773909]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773909/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773909/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3e64540df12740adabd8977b313f3949

    Command failed on smithi052 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    1002c4c8-b2ce-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.52 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773914]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:34
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773914/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773914/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c13a4f7432794c4aaacb95b8f0291069

    Command failed on smithi061 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    9af66eb8-b2ce-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.61 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773916]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:41:18
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773916/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773916/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b8e432de6e745c192761084a7ac206e

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi145 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6773928]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:48
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773928/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773928/
sentry: https://sentry.ceph.com/organizations/ceph/?query=664b785a137f4d74b29c2d8eb9442b83

    Command failed on smithi017 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    7ebc321c-b2d0-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.17 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773933]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773933/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773933/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5f21712726924f1582b513c54cd72d97

    Command failed on smithi077 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    6f8e6e76-b2d1-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.77 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773943]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
-----------------------------------------------------------------
time:   00:12:51
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773943/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773943/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f70106bab16439abe5cab674f544db3

    Command failed on smithi123 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    1564a130-b2d2-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.123 --skip-admin-label
    && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773947]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773947/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773947/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a7f8f1586704598ae9d23170ee0c602

    Command failed on smithi083 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    90b932ce-b2d2-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.83 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773958]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:41:44
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773958/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773958/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3dbafc08831045cf94a974724fdc81e0

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi022 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6773959]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:10:34
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773959/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773959/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c08fb3acfd87460994d1a89393c8c57d

    Command failed on smithi017 with status 1: "(cd /home/ubuntu/cephtest &&
    exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-
    mnt.admin sudo adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f
    --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok'
    /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)"

[6773960]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:15
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773960/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773960/
sentry: https://sentry.ceph.com/organizations/ceph/?query=784ca2370f324c0e822e91d1db573bef

    Command failed on smithi050 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    0fde98d6-b2d4-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.50 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773967]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773967/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773967/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e78df3b477b14b718038dde931004aa1

    Command failed on smithi100 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    07433c94-b2d5-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.100 --skip-admin-label
    && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773976]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}}
-----------------------------------------------------------------
time:   00:12:39
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773976/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773976/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e958cdc05dc24b47a5d75ec5c2e47016

    Command failed on smithi160 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    7d6d6066-b2d5-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.160 --skip-admin-label
    && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773987]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773987/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773987/
sentry: https://sentry.ceph.com/organizations/ceph/?query=483d0515bb134b3c8d9c679b06cc8b31

    Command failed on smithi084 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    b804b3e0-b2d6-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.84 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6773996]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}}
-----------------------------------------------------------------
time:   00:12:48
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773996/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773996/
sentry: https://sentry.ceph.com/organizations/ceph/?query=880c69a82c4a463db62eec8ad3843415

    Command failed on smithi092 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    3e3a6af4-b2d7-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.92 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774003]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774003/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774003/
sentry: https://sentry.ceph.com/organizations/ceph/?query=42e30c60afcd4e2696e08e6c2f776e73

    Command failed on smithi120 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    198791a4-b2d8-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.120 --skip-admin-label
    && sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774009]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:13:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774009/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774009/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4391366a0d9c4c6397c0620bfb18683b

    Command failed on smithi165 with status 127: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    cd52e30a-b2d8-11ec-8c36-001a4aab830c -- ceph mon dump -f json'

[6774013]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
-----------------------------------------------------------------
time:   00:12:46
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774013/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774013/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7dffe9e97c5a43b59798e353773d43a8

    Command failed on smithi082 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    2b6b7902-b2d9-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.82 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774017]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774017/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774017/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0f06ad05eb5c4ce58915f9be4480b83f

    Command failed on smithi008 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    cce7a652-b2d9-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774020]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:20:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774020/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774020/
sentry: https://sentry.ceph.com/organizations/ceph/?query=715088fb26c34ba2a78bb96297356fc6

    reached maximum tries (90) after waiting for 540 seconds

[6774029]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
-----------------------------------------------------------------
time:   00:12:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774029/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774029/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c0333778a6a54ee7b061b4c9b546a9c6

    Command failed on smithi074 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    635aeec2-b2db-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.74 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774033]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774033/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774033/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1461b5a4973d4e7f93f9dff22dc28816

    Command failed on smithi053 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    f29a7486-b2db-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774052]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
-----------------------------------------------------------------
time:   00:17:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774052/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774052/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a8d3fe88ce1a4e439688469876ddf64c

    Command failed on smithi074 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    93add79e-b2de-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.74 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774056]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:29:38
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774056/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774056/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1b4bf46aad67405a9d62b60589698935

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi099 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6774058]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774058/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774058/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f5f0d1b73ccc42f48f6b442497eba3df

    Command failed on smithi092 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    93ffb004-b2df-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.92 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774067]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}}
-----------------------------------------------------------------
time:   00:12:59
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774067/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774067/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ff7bc648d1c4bfbb655e2922c248f5c

    Command failed on smithi059 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    6fd9a742-b2e0-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.59 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774071]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics}
-----------------------------------------------------------------
time:   00:20:18
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774071/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=93a631ec70f243d7b8853f958c90116b

    Test failure: test_perf_stats_stale_metrics
    (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

[6774072]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:47
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774072/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774072/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b5b660dd1c704300b76ce546f696e0d6

    Command failed on smithi007 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    7ece1052-b2e1-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774073]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:23:03
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774073/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774073/

    "2022-04-03T00:06:44.887013+0000 osd.1 (osd.1) 14 : cluster [ERR] 5.19s0
    deep-scrub : stat mismatch, got 0/26 objects, 0/0 clones, 0/26 dirty, 0/0
    omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/34679 bytes, 0/0
    manifest objects, 0/0 hit_set_archive bytes." in cluster log

[6774082]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:09
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774082/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774082/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d3d8800a3b14df6846c46fb9b6b0aec

    Command failed on smithi008 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    32159068-b2e2-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774086]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774086/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774086/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3833fa1a6bed455eb0e5b387c5e66b4c

    Command failed on smithi092 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    f75acad2-b2e2-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.92 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774090]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:20:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774090/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774090/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a941c1ac5f94c66895a31f03d58d961

    reached maximum tries (90) after waiting for 540 seconds

[6774100]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774100/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774100/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e7591ea012a416f890ef250c1828907

    Command failed on smithi025 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    d8e4af6c-b2e4-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.25 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774110]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774110/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774110/
sentry: https://sentry.ceph.com/organizations/ceph/?query=70586812e9f84e00b77b667f93f1ada3

    Command failed on smithi053 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    50a9cd42-b2e6-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774119]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774119/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774119/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5379532475f74ae6856aad43c8c17aaa

    Command failed on smithi027 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    34f8d402-b2e7-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774124]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774124/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774124/
sentry: https://sentry.ceph.com/organizations/ceph/?query=06949d3f8c1e458fa653cb51bebdb70c

    Command failed on smithi025 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    46034790-b2e8-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.25 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774133]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:30
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774133/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774133/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b1fe6734696a4221af2dea74985d03e6

    Command failed on smithi053 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    8ce54fb8-b2e9-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774134]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:49:53
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774134/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774134/

    "2022-04-03T01:02:17.231716+0000 mon.a (mon.0) 479 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6774140]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:23:31
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774140/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774140/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d1a3f30e37f14467a6feef66571135cc

    Command failed (workunit test suites/fsstress.sh) on smithi073 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6774142]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774142/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774142/
sentry: https://sentry.ceph.com/organizations/ceph/?query=51126eac63674cc58ef1fc2f813c6081

    Command failed on smithi083 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    463841e0-b2eb-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.83 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774148]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:26:17
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774148/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774148/
sentry: https://sentry.ceph.com/organizations/ceph/?query=020a79b6d4ee44e9afc2f77cbd2529bd

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi133 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6774151]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774151/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774151/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e6408aa208e34e75a4a466d9c1afe555

    Command failed on smithi055 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    97530294-b2ec-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.55 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774156]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774156/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774156/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d739d4489454cb0bba282ac7f239545

    Command failed on smithi039 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    d0fd6916-b2ed-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.39 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774173]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774173/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774173/
sentry: https://sentry.ceph.com/organizations/ceph/?query=49cd0e66feaa41d4b03f162170e4ac49

    Command failed on smithi007 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    ecab368c-b2ef-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774176]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:21:41
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774176/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774176/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b33fae24c8f7435cb322097493d9776c

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi006 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6774177]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774177/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774177/
sentry: https://sentry.ceph.com/organizations/ceph/?query=129160b434c048819fded026f84b91e0

    Command failed on smithi005 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    9331bada-b2f0-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.5 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774190]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:03
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774190/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774190/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8cb188efac284450bdb73b51fb054746

    Command failed on smithi002 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    a4d7ff22-b2f2-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774193]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
-----------------------------------------------------------------
time:   00:13:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774193/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774193/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2592db59359445298f3a215d3d30f2a

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

[6774194]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774194/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774194/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a9f6fbc3e5cf4e378b051b408975cc45

    Command failed on smithi053 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    3944c0fa-b2f3-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774196]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:14:40
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774196/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774196/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01fa0e78316f40b39c4df7b8f89de9c2

[6774202]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:37:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774202/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774202/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d4aec30a86f74a72a04415b526fba718

    Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on
    smithi018 with status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-
    split.sh'

[6774204]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
-----------------------------------------------------------------
time:   00:13:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774204/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774204/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2fa73422c88445c7afb2f4e084c5c4d9

    Command failed on smithi007 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    de66cd8e-b2f4-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'

[6774208]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
-----------------------------------------------------------------
time:   00:13:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774208/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774208/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1ecdf85abfc64dd1acd2bb63169fd01a

    Command failed on smithi052 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-
    ci/ceph:cc9ed509562d767c1a5d8fc115593bc1453dfea1 -v bootstrap --fsid
    f2a05bd4-b2f5-11ec-8c36-001a4aab830c --config
    /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf
    --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key
    /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-
    daemons --skip-monitoring-stack --mon-ip 172.21.15.52 --skip-admin-label &&
    sudo chmod +r /etc/ceph/ceph.client.admin.keyring'



Dead
=================================================================
[6773878]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:36:40
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773878/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773878/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1bf384763b624ee69868a89cf7e4ff4e

    Command failed (workunit test suites/fsstress.sh) on smithi143 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6773931]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773931/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773931/

    hit max job timeout

[6773944]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773944/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773944/

    hit max job timeout

[6773949]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773949/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773949/

    hit max job timeout

[6773988]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773988/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773988/

    hit max job timeout

[6774011]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774011/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774011/

    hit max job timeout

[6774036]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774036/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774036/

    hit max job timeout

[6774037]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774037/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774037/

    hit max job timeout

[6774069]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774069/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774069/

    hit max job timeout

[6774080]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774080/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774080/

    hit max job timeout

[6774091]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774091/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774091/

    hit max job timeout

[6774132]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774132/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774132/

    hit max job timeout

[6774163]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774163/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774163/

    hit max job timeout

[6774167]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774167/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774167/

    hit max job timeout

[6774172]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:36:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774172/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774172/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f37d3c8a391f473c812a5460f93824db

    Command failed (workunit test suites/fsstress.sh) on smithi184 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

[6774191]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774191/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774191/

    hit max job timeout

[6774198]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   03:36:36
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774198/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774198/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ef4a3ff122f5407facdb9371da8a981f

    Command failed (workunit test suites/fsstress.sh) on smithi082 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=cc9ed509562d767c1a5d8fc115593bc1453dfea1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'



Running
=================================================================
[6773879] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773879/

[6773970] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773970/

[6773983] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773983/

[6773984] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773984/

[6773999] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773999/

[6774024] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774024/

[6774079] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774079/

[6774111] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774111/

[6774122] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774122/

[6774161] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774161/

[6774212] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774212/



Pass
=================================================================
[6773842] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:00:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773842/

[6773843] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:16:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773843/

[6773844] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:01
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773844/

[6773847] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:20:24
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773847/

[6773848] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:42
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773848/

[6773849] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773849/

[6773850] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137}
time:   00:14:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773850/

[6773852] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full}
time:   00:16:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773852/

[6773853] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:15:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773853/

[6773854] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror}
time:   02:09:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773854/

[6773855] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit}
time:   00:33:39
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773855/

[6773856] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:24:52
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773856/

[6773857] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:51:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773857/

[6773858] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:17:01
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773858/

[6773859] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:40:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773859/

[6773860] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:03:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773860/

[6773861] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:16:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773861/

[6773862] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:14:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773862/

[6773863] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773863/

[6773864] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:21:47
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773864/

[6773865] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   00:51:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773865/

[6773866] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
time:   00:20:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773866/

[6773869] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots}
time:   00:30:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773869/

[6773870] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:54:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773870/

[6773871] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:28:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773871/

[6773872] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:19:59
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773872/

[6773873] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:15:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773873/

[6773874] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773874/

[6773875] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}
time:   00:26:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773875/

[6773877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:09
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773877/

[6773880] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:55:52
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773880/

[6773882] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:16:01
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773882/

[6773883] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration}
time:   00:18:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773883/

[6773884] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:15:57
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773884/

[6773885] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   01:16:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773885/

[6773886] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:16:55
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773886/

[6773887] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773887/

[6773888] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay}
time:   00:13:00
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773888/

[6773891] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}}
time:   00:13:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773891/

[6773892] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:01:15
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773892/

[6773893] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:15:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773893/

[6773894] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:56:39
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773894/

[6773895] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   01:09:09
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773895/

[6773896] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773896/

[6773898] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries}
time:   00:13:41
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773898/

[6773899] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:19:18
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773899/

[6773900] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773900/

[6773901] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}}
time:   02:57:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773901/

[6773902] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:36:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773902/

[6773904] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:22:42
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773904/

[6773905] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota}
time:   00:15:42
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773905/

[6773906] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:54:46
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773906/

[6773907] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773907/

[6773908] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:40
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773908/

[6773910] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:21:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773910/

[6773911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:16:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773911/

[6773912] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:17:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773912/

[6773913] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:17:38
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773913/

[6773915] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:42
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773915/

[6773917] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:17:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773917/

[6773918] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
time:   00:46:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773918/

[6773919] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:36:34
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773919/

[6773920] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:15:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773920/

[6773921] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:49:24
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773921/

[6773922] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:13:41
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773922/

[6773923] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:22:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773923/

[6773924] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:51:39
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773924/

[6773925] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
time:   01:55:15
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773925/

[6773926] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}}
time:   00:20:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773926/

[6773927] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:53
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773927/

[6773929] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773929/

[6773930] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:46
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773930/

[6773932] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:13:48
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773932/

[6773934] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:10:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773934/

[6773935] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773935/

[6773936] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:28:44
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773936/

[6773937] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:16:36
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773937/

[6773938] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:25:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773938/

[6773939] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:25:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773939/

[6773940] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:19:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773940/

[6773941] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:15:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773941/

[6773942] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:18:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773942/

[6773945] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:12:26
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773945/

[6773946] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:58:18
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773946/

[6773948] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:21:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773948/

[6773950] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:13:30
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773950/

[6773951] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773951/

[6773952] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb}
time:   00:15:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773952/

[6773953] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773953/

[6773954] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:18:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773954/

[6773955] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}}
time:   00:24:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773955/

[6773956] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace}
time:   00:13:37
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773956/

[6773957] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773957/

[6773961] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:49:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773961/

[6773962] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:19:57
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773962/

[6773963] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:16:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773963/

[6773964] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush}
time:   00:14:32
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773964/

[6773965] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773965/

[6773966] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:13:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773966/

[6773968] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:19:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773968/

[6773969] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773969/

[6773971] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits}
time:   00:18:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773971/

[6773972] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}}
time:   00:24:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773972/

[6773973] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:10:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773973/

[6773974] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:16:24
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773974/

[6773975] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:17:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773975/

[6773977] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:35:46
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773977/

[6773978] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:21:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773978/

[6773979] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead}
time:   00:12:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773979/

[6773980] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:15:47
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773980/

[6773981] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:14:51
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773981/

[6773982] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:51:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773982/

[6773985] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/lockdep}
time:   00:47:51
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773985/

[6773986] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:59:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773986/

[6773989] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
time:   00:39:59
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773989/

[6773990] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:06:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773990/

[6773991] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
time:   00:41:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773991/

[6773992] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:23:31
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773992/

[6773993] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:13:01
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773993/

[6773994] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:23:39
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773994/

[6773995] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:20:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773995/

[6773997] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage}
time:   00:19:49
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773997/

[6773998] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:14:42
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6773998/

[6774000] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774000/

[6774001] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:08
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774001/

[6774002] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:58:19
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774002/

[6774004] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:16:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774004/

[6774005] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan}
time:   00:30:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774005/

[6774006] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774006/

[6774007] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:32:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774007/

[6774008] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:19:09
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774008/

[6774010] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774010/

[6774012] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports}
time:   00:51:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774012/

[6774014] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:56:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774014/

[6774015] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:17:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774015/

[6774016] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:56
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774016/

[6774018] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:26:36
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774018/

[6774019] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
time:   00:15:17
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774019/

[6774021] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:28:21
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774021/

[6774022] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:25:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774022/

[6774023] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}}
time:   00:25:59
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774023/

[6774025] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:28:36
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774025/

[6774026] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:27:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774026/

[6774027] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:58:03
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774027/

[6774028] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment}
time:   00:19:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774028/

[6774030] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:56
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774030/

[6774031] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:12:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774031/

[6774032] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:21:17
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774032/

[6774034] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair}
time:   00:21:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774034/

[6774035] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:14:47
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774035/

[6774038] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:52:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774038/

[6774039] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:33:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774039/

[6774040] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:16:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774040/

[6774041] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774041/

[6774042] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/whitelist_health}
time:   00:22:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774042/

[6774043] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}}
time:   00:17:19
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774043/

[6774044] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:25:17
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774044/

[6774045] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:31:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774045/

[6774046] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:30
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774046/

[6774047] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:37:30
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774047/

[6774048] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind}
time:   01:59:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774048/

[6774049] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}}
time:   00:48:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774049/

[6774050] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush}
time:   00:11:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774050/

[6774051] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774051/

[6774053] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:21:32
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774053/

[6774054] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:18
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774054/

[6774055] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:37
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774055/

[6774057] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
time:   00:25:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774057/

[6774059] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:06:26
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774059/

[6774060] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:17:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774060/

[6774061] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:16:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774061/

[6774062] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:23:35
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774062/

[6774063] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:23:32
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774063/

[6774064] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry}
time:   00:12:41
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774064/

[6774065] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:15:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774065/

[6774066] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:16:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774066/

[6774068] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}}
time:   00:13:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774068/

[6774070] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:10:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774070/

[6774074] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:14:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774074/

[6774075] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774075/

[6774076] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   00:50:01
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774076/

[6774077] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:03
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774077/

[6774078] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc}
time:   00:19:55
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774078/

[6774081] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:15:34
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774081/

[6774083] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:16:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774083/

[6774084] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable}
time:   00:15:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774084/

[6774085] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:15:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774085/

[6774087] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:20:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774087/

[6774088] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774088/

[6774089] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:15:09
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774089/

[6774092] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}}
time:   01:27:31
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774092/

[6774093] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm}
time:   00:12:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774093/

[6774094] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   00:59:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774094/

[6774095] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:44:31
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774095/

[6774096] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:58:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774096/

[6774097] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:23:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774097/

[6774098] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:17:34
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774098/

[6774099] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:16:19
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774099/

[6774101] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:22:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774101/

[6774102] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota}
time:   00:15:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774102/

[6774103] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:15:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774103/

[6774104] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:13:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774104/

[6774105] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:48:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774105/

[6774106] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:35:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774106/

[6774107] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
time:   00:15:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774107/

[6774108] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/lockdep}
time:   01:05:41
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774108/

[6774109] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}}
time:   00:17:03
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774109/

[6774112] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:57:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774112/

[6774113] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:39:08
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774113/

[6774114] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs}
time:   00:14:44
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774114/

[6774115] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:34:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774115/

[6774116] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:24:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774116/

[6774117] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:13:16
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774117/

[6774118] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:18:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774118/

[6774120] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:14:33
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774120/

[6774121] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub}
time:   00:20:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774121/

[6774123] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:53:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774123/

[6774125] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:16:57
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774125/

[6774126] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:28:48
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774126/

[6774127] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:15:44
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774127/

[6774128] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap}
time:   00:16:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774128/

[6774129] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}}
time:   00:34:12
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774129/

[6774130] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:50
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774130/

[6774131] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774131/

[6774135] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:57:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774135/

[6774136] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:35
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774136/

[6774137] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python}
time:   00:15:44
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774137/

[6774138] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:47
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774138/

[6774139] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:04
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774139/

[6774141] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774141/

[6774143] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:25:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774143/

[6774144] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots}
time:   00:22:01
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774144/

[6774145] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:14:37
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774145/

[6774146] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   01:08:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774146/

[6774147] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
time:   00:24:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774147/

[6774149] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   01:12:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774149/

[6774150] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays}
time:   00:31:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774150/

[6774152] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:10
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774152/

[6774153] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:55
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774153/

[6774154] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:15:05
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774154/

[6774155] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:18:18
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774155/

[6774157] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:17:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774157/

[6774158] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration}
time:   00:20:43
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774158/

[6774159] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:30:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774159/

[6774160] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   01:12:09
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774160/

[6774162] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:17:30
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774162/

[6774164] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:15:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774164/

[6774165] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest}
time:   00:17:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774165/

[6774166] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:23:23
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774166/

[6774168] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/valgrind}
time:   02:03:02
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774168/

[6774169] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}}
time:   00:22:45
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774169/

[6774170] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:16:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774170/

[6774171] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay}
time:   00:12:29
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774171/

[6774174] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:17:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774174/

[6774175] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:28
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774175/

[6774178] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:04:14
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774178/

[6774179] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries}
time:   00:16:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774179/

[6774180] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:15:51
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774180/

[6774181] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:14:25
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774181/

[6774182] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}}
time:   00:24:20
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774182/

[6774183] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774183/

[6774184] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:17:26
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774184/

[6774185] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:20:55
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774185/

[6774186] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:23:44
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774186/

[6774187] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota}
time:   00:17:07
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774187/

[6774188] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:18:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774188/

[6774189] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:16:32
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774189/

[6774192] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:55:27
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774192/

[6774195] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:16:46
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774195/

[6774197] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:17:37
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774197/

[6774199] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:23:39
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774199/

[6774200] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:06
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774200/

[6774201] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:57:30
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774201/

[6774203] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   01:16:22
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774203/

[6774205] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:14:58
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774205/

[6774206] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:14:08
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774206/

[6774207] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:13:54
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774207/

[6774209] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:19:42
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774209/

[6774210] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:36
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774210/

[6774211] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:30:13
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774211/

[6774213] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   00:23:48
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774213/

[6774214] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   01:05:11
info:   http://pulpito.ceph.com/vshankar-2022-04-02_19:34:47-fs-wip-55110-for-testing-testing-default-smithi/6774214/