Found 64 jobs Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Ignoring malparsed backtrace: ceph version 17.0.0-1902-g88e3ea08 (88e3ea0854f69cc01782866be7a1c31166ea1c79) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f459ccebd2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7f459ccebf33] Ignoring malparsed backtrace: 2021-03-11T17:27:13.183 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 2021 , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: - , 2021-03-11T17:27:13.184 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:12 smithi059 conmon[31137]: 03 Ignoring malparsed backtrace: 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: audit 2021-03- , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: 11T17:27:14.083319+0000 mon.a , 2021-03-11T17:27:15.294 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:14 smithi098 conmon[25960]: (mon.0) 2167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: :27:14.864863+0000 mon.a (mon.0) 2168 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:27:16.098 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:15 smithi059 conmon[31137]: -11T17:27:14.866222+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: -03-11T17:27: , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: 20.616277+0000 osd.3 (osd , 2021-03-11T17:27:21.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:20 smithi059 conmon[31137]: .3) 219 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:27:24.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.4 dump_historic_ops , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: cluster , 2021-03-11T17:27:25.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:24 smithi059 conmon[31137]: 2021-03 Ignoring malparsed backtrace: 2021-03-11T17:27:26.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:25 smithi098 conmon[25960]: .0) 2185 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:27:26.864 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:26 smithi059 conmon[31137]: cluster 2021-03-11T17:27:25.868918+0000 Ignoring malparsed backtrace: 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:27.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:19:55.613227+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", , 2021-03-11T17:27:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.631839+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:20.659531+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:27.573 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6229999999999997e-06 , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016994199999999999 , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.064195426 , 2021-03-11T17:27:27.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.696 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030461099999999999 , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:27.723 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.814729+0000", , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088126700000000003 , 2021-03-11T17:27:27.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.821980+0000", , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2696e-05 , 2021-03-11T17:27:27.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.824401+0000", , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040305300000000001 , 2021-03-11T17:27:27.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:27.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:25:55.875313+0000", , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.053224159 , 2021-03-11T17:27:27.829 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:28.942 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: T17:27:27.909207+0000 mgr.y (mgr. , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: 14144) 981 : cephadm [INF] Refreshing smithi098 facts , 2021-03-11T17:27:28.943 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:28 smithi059 conmon[31137]: cephadm 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: (osd.3) 220 : cluster [DBG] 4.9 scrub ok , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: audit , 2021-03-11T17:27:30.619 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:30 smithi059 conmon[25665]: 2021-03-11T17:27:29.544227+0000 mon.a ( Ignoring malparsed backtrace: 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: osd.5) 113 : cluster [DBG] 6.d scrub ok , 2021-03-11T17:27:33.539 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:27:33.540 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:33 smithi098 conmon[25960]: T17:27:32.219946+0000 osd.7 (osd Ignoring malparsed backtrace: 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster [DBG] pgmap v1655: 168 pgs: 1 active+remapped+backfill_toofull, 157 active+clean, 1 active+clean+remapped, 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 823 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2106 objects misplaced (0.997%); 278 B/s, 3 objects/s recovering , 2021-03-11T17:27:33.897 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: , 2021-03-11T17:27:33.898 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:33 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:36.549 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[25665]: 03-11T17:27:35.816377+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: audit 2021-03-11T17:27:34.879088+0000 mgr.y (mgr. , 2021-03-11T17:27:36.550 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:36 smithi059 conmon[31137]: 14144) 987 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:27:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.745913+0000", , 2021-03-11T17:27:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018925882000000002 Ignoring malparsed backtrace: 2021-03-11T17:27:37.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.764903+0000", , 2021-03-11T17:27:37.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014813517 Ignoring malparsed backtrace: 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:24.779905+0000", , 2021-03-11T17:27:37.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1248999999999995e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:37.612 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.75102276999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0350574619999999, , 2021-03-11T17:27:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.645 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 253.738738641, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0230874999999999, , 2021-03-11T17:27:37.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:37.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:37.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7296 Ignoring malparsed backtrace: 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:23:23.757961+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:16.793675+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.802850+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", , 2021-03-11T17:27:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:18.810106+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.805789+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897832+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.897812+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961390+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", , 2021-03-11T17:27:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:54.961486+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:37.992 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", , 2021-03-11T17:27:37.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:58.802059+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.14769", , 2021-03-11T17:27:38.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", Ignoring malparsed backtrace: 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: 27:37.324026+0000 osd.7 , 2021-03-11T17:27:38.620 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: (osd.7) 118 : cluster , 2021-03-11T17:27:38.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:38 smithi059 conmon[25665]: [DBG] 6.15 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27 , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: :40.795989+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch , 2021-03-11T17:27:41.794 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:41 smithi098 conmon[25960]: audit 2021-03-11T17:27:40.819218 Ignoring malparsed backtrace: 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: osd.3 (osd.3) 224 : cluster [DBG] 3.18 scrub ok , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: cluster 2021- , 2021-03-11T17:27:45.544 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:45 smithi098 conmon[25960]: 03-11T17:27:44.162302+0000 osd.2 (osd.2) 192 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: [DBG] , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: , 2021-03-11T17:27:45.641 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:45 smithi059 conmon[25665]: 6.19 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:27:46.623 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: : audit [INF] from='client.? 172.21.15.59:0/1579849594' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: cluster 2021-03-11T17:27:45.331731+0000 mon.a , 2021-03-11T17:27:46.624 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:46 smithi059 conmon[25665]: (mon.0) 2210 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in Ignoring malparsed backtrace: 2021-03-11T17:27:47.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:17.067471+0000", , 2021-03-11T17:27:47.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.041203327999999997 Ignoring malparsed backtrace: 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 386.15519031100001, , 2021-03-11T17:27:47.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035657427999999998, , 2021-03-11T17:27:47.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { Ignoring malparsed backtrace: 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:21.634409+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.915 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:47.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:21:47.747038+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:47.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672801+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.967 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting_for_map not empty", , 2021-03-11T17:27:47.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:22:45.672935+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:47.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", , 2021-03-11T17:27:47.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.801372+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.021 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:48.070 INFO:teuthology.orchestra.run.smithi098.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4851000000000001e-05 , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:48.099 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:48.122 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.14769.0:9488 7.7 7.1db2c917 (undecoded) ondisk+write+known_if_redirected e943)", , 2021-03-11T17:27:48.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:26:55.006665+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:26:55.006762+0000", , 2021-03-11T17:27:48.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1202999999999999e-05 , 2021-03-11T17:27:48.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.59:0/582871736", , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 9582 , 2021-03-11T17:27:48.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:48.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829079+0000", , 2021-03-11T17:27:48.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9199999999999998e-06 Ignoring malparsed backtrace: 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:00.829114+0000", , 2021-03-11T17:27:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030738900000000001 Ignoring malparsed backtrace: 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: - , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: 03 , 2021-03-11T17:27:49.870 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:49 smithi059 conmon[31137]: -11T17:27:48.128797+0000 osd.0 (osd.0) 178 : Ignoring malparsed backtrace: 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster [DBG] , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: 6.29 scrub ok , 2021-03-11T17:27:51.638 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:51.683 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: 996 : cluster , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: [DBG] pgmap v1672: 168 pgs: 1 active+remapped+backfill_toofull, 160 active+clean, 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait; 801 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21/2067 objects misplaced (1.016%) , 2021-03-11T17:27:51.684 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:27:51 smithi059 conmon[31137]: Ignoring malparsed backtrace: 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:52.908334+0000 , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: osd.5 (osd.5) 119 : cluster [DBG] 3.16 deep-scrub ok , 2021-03-11T17:27:55.131 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:54 smithi059 conmon[25665]: cluster 2021-03-11T17:27:53.121690 Ignoring malparsed backtrace: 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 184834+0000 osd.2 (osd.2) , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: 195 : cluster [DBG] 5.4 scrub ok , 2021-03-11T17:27:58.128 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:27:57 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:5239 7.8 e530/105 7:117dc0ad:::smithi05964533-7:head v 530'760, mlcod=529'756)", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2021-03-11T17:19:56.749960+0000", , 2021-03-11T17:27:58.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 481.57735029000003, Ignoring malparsed backtrace: 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4037e-05 , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.374 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:20:44.669582+0000", , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1620000000000002e-06 , 2021-03-11T17:27:58.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.14769.0:7273 7.c e733/187 7:3c4e5647:::smithi05964533-26:head v 733'776, mlcod=729'773)", Ignoring malparsed backtrace: 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:09.758716+0000", , 2021-03-11T17:27:58.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1605999999999999e-05 Ignoring malparsed backtrace: 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", , 2021-03-11T17:27:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:24:32.177303+0000", Ignoring malparsed backtrace: 2021-03-11T17:27:58.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] , 2021-03-11T17:27:58.534 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: { , 2021-03-11T17:27:58.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1050000000000003e-06 , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, , 2021-03-11T17:27:58.584 INFO:teuthology.orchestra.run.smithi098.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011617951 , 2021-03-11T17:27:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: } , 2021-03-11T17:27:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] Ignoring malparsed backtrace: 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:26.018518+0000", , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109892 , 2021-03-11T17:27:58.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2021-03-11T17:27:55.892011+0000", , 2021-03-11T17:27:58.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-03-11T17:27:59.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight , 2021-03-11T17:28:00.065 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: cluster 2021-03-11T17:27: , 2021-03-11T17:28:00.093 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:27:59 smithi098 conmon[25960]: 57.728901+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.13 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[31137]: (mon.0) 2227 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: cluster 2021-03-11T , 2021-03-11T17:28:03.869 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:03 smithi059 conmon[25665]: 17:28:02.135411+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.c scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:06.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: osd.2) 199 : cluster [DBG] 6.e scrub ok , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: cluster 2021-03-11 , 2021-03-11T17:28:06.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:05 smithi098 conmon[25960]: T17:28:04.885240+0000 osd.5 (osd.5 Ignoring malparsed backtrace: 2021-03-11T17:28:07.066 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: 3.1d scrub ok , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:07.067 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:06 smithi098 conmon[25960]: -03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.161957+0000 osd.0 ( , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: osd.0) 184 : cluster [DBG] 7.6 scrub ok , 2021-03-11T17:28:11.116 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:10 smithi059 conmon[25665]: cluster 2021-03-11T17:28:09.899118+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: -03-11T17:28:12.897170+0000 osd.5 (osd.5) 134 , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[25665]: : cluster [DBG] 6.2c scrub ok , 2021-03-11T17:28:14.054 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:13 smithi059 conmon[31137]: cluster 2021-03-11T17:28:12 Ignoring malparsed backtrace: 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", , 2021-03-11T17:28:14.326 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:22:45.698272+0000", , 2021-03-11T17:28:14.327 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 1.0693000000000001e-05 Ignoring malparsed backtrace: 2021-03-11T17:28:14.350 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.351 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "reached_pg", , 2021-03-11T17:28:14.395 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:23:24.766334+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.475 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.476 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "header_read", Ignoring malparsed backtrace: 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: ] , 2021-03-11T17:28:14.518 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.000108533 , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.543 INFO:teuthology.orchestra.run.smithi059.stdout: { Ignoring malparsed backtrace: 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.821578+0000", , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.00090540800000000002 , 2021-03-11T17:28:14.567 INFO:teuthology.orchestra.run.smithi059.stdout: }, Ignoring malparsed backtrace: 2021-03-11T17:28:14.591 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.871993+0000", , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.048196946999999997 , 2021-03-11T17:28:14.592 INFO:teuthology.orchestra.run.smithi059.stdout: } Ignoring malparsed backtrace: 2021-03-11T17:28:14.615 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.834624+0000", , 2021-03-11T17:28:14.618 INFO:teuthology.orchestra.run.smithi059.stdout: "duration": 0.038369894000000002 Ignoring malparsed backtrace: 2021-03-11T17:28:14.644 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "throttled", , 2021-03-11T17:28:14.658 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.839591+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "done", , 2021-03-11T17:28:14.699 INFO:teuthology.orchestra.run.smithi059.stdout: "time": "2021-03-11T17:25:55.875525+0000", Ignoring malparsed backtrace: 2021-03-11T17:28:14.903 INFO:teuthology.orchestra.run.smithi059.stdout: }, , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: { , 2021-03-11T17:28:14.904 INFO:teuthology.orchestra.run.smithi059.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: audit , 2021-03-11T17:28:16.395 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: , 2021-03-11T17:28:16.396 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:15 smithi059 conmon[31137]: 2021-03-11T17 Ignoring malparsed backtrace: 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: 2) 200 : cluster [DBG] 3.f deep-scrub ok , 2021-03-11T17:28:20.375 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: audit 2021-03 , 2021-03-11T17:28:20.376 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:19 smithi059 conmon[25665]: -11T17:28:19.380741+0000 mon.c (mon Ignoring malparsed backtrace: 2021-03-11T17:28:21.393 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: osd.3) 236 : cluster [DBG] purged_snaps scrub ok , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: cluster 2021-03 , 2021-03-11T17:28:21.394 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:20 smithi059 conmon[31137]: -11T17:28:20.848554+0000 mon.a Ignoring malparsed backtrace: 2021-03-11T17:28:25.128 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: mon.a (mon.0) 2246 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: , 2021-03-11T17:28:25.129 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:24 smithi059 conmon[31137]: cluster 2021-03-11T17:28 Ignoring malparsed backtrace: 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: mon.a (mon.0) 2247 : , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in , 2021-03-11T17:28:26.841 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:26 smithi059 conmon[31137]: cluster 2021 Ignoring malparsed backtrace: 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: -11T17:28:27.869873+0000 , 2021-03-11T17:28:29.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:28 smithi098 conmon[25960]: mon.a (mon.0) 2251 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:28:30.865 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: cluster 2021-03-11T17:28: , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[31137]: 29.198957+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.5 scrub ok , 2021-03-11T17:28:30.866 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:30 smithi059 conmon[25665]: cluster 2021-03-11T17: Ignoring malparsed backtrace: 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[25665]: 0) 2259 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: cluster 2021-03-11T17 , 2021-03-11T17:28:33.873 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:33 smithi059 conmon[31137]: :28:31.769993+0000 osd.4 (osd.4) 222 : cluster [DBG] 6.32 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: ) 1025 : cluster [DBG] pgmap v1717: 168 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 5 peering, 161 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 21/2004 objects misplaced (1.048%) , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: cluster 2021-03-11T17:28:34 , 2021-03-11T17:28:36.043 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:35 smithi098 conmon[25960]: .596823+0000 mon.a (mon.0) 2261 : Ignoring malparsed backtrace: 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: 6.2c scrub ok , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: cluster 2021 , 2021-03-11T17:28:38.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:37 smithi098 conmon[25960]: -03-11T17:28:36. Ignoring malparsed backtrace: 2021-03-11T17:28:41.125 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: . , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: 5 , 2021-03-11T17:28:41.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:40 smithi059 conmon[25665]: ) Ignoring malparsed backtrace: 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: cluster 2021-03-11T17:28:42 , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: .002334+0000 osd.5 (osd.5) 147 : cluster [DBG] , 2021-03-11T17:28:43.122 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:42 smithi059 conmon[31137]: 5.7 scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:45.371 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: 2021-03-11T17:28 , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: :44.638671+0000 mon.a , 2021-03-11T17:28:45.372 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:44 smithi059 conmon[25665]: (mon.0) 2273 : audit [INF] Ignoring malparsed backtrace: 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: cluster 2021 , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: -03-11T17:28:46.581317+0000 mgr.y , 2021-03-11T17:28:47.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:46 smithi059 conmon[31137]: (mgr.14144) 1033 : cluster [DBG] pgmap v1731: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail Ignoring malparsed backtrace: 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: [DBG] purged_snaps scrub starts , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: cluster 2021-03-11T17:28:50.036542+0000 , 2021-03-11T17:28:51.543 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:51 smithi098 conmon[25960]: osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub ok Ignoring malparsed backtrace: 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: pgmap v1737: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s , 2021-03-11T17:28:53.373 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: audit , 2021-03-11T17:28:53.374 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:53 smithi059 conmon[31137]: 2021-03-11T17:28:52.900067+0000 mon.a (mon.0) Ignoring malparsed backtrace: 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1740: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 966 B/s rd, 0 op/s , 2021-03-11T17:28:57.621 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: audit 2021 , 2021-03-11T17:28:57.622 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:28:57 smithi059 conmon[25665]: -03-11T17:28:56.750818+0000 Ignoring malparsed backtrace: 2021-03-11T17:28:59.619 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:28:59 smithi059 conmon[31137]: .14144) 1040 : cluster [DBG] pgmap v1742: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 132 B/s rd, 0 op/s , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: cluster 2021-03-11T17:28:58.089512+0000 osd.5 (osd.5) 154 : cluster , 2021-03-11T17:28:59.792 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:28:59 smithi098 conmon[25960]: [DBG] 6.d deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: -03-11T17 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: :29:02.329725 , 2021-03-11T17:29:03.576 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:03 smithi059 conmon[25665]: +0000 osd.2 (osd.2) Ignoring malparsed backtrace: 2021-03-11T17:29:05.798 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:05 smithi098 conmon[25960]: , 2021-03-11T17:29:06.616 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: audit 2021-03-11T17:29:04. , 2021-03-11T17:29:06.617 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:06 smithi059 conmon[31137]: 968787+0000 mgr.y (mgr.14144) 1044 : audit [DBG] from='client.34379 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch Ignoring malparsed backtrace: 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: [DBG] 3.2 scrub ok , 2021-03-11T17:29:09.431 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: cluster 2021-03-11T17:29:08.591865 , 2021-03-11T17:29:09.432 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:09 smithi059 conmon[31137]: +0000 mgr.y (mgr.14144) 1046 : cluster [DBG] pgmap v1750: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 663 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:12.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell --fsid 946db372-828b-11eb-9083-001a4aab830c -- ceph tell osd.4 flush_pg_stats , Inferring config /var/lib/ceph/946db372-828b-11eb-9083-001a4aab830c/mon.a/config , 2021-03-11T17:29:12.947 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:12 smithi059 conmon[25665]: cluster 2021-03-11T17:29:12. Ignoring malparsed backtrace: 2021-03-11T17:29:16.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: cluster 2021-03-11T , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: 17:29:14.644676+0000 osd.4 (osd.4 , 2021-03-11T17:29:16.120 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:15 smithi059 conmon[31137]: ) 232 : cluster [DBG] 3.1a scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:19.295 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: -11T17:29:18.597635 , 2021-03-11T17:29:19.296 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:18 smithi098 conmon[25960]: +0000 mgr.y (mgr.14144) 1052 : cluster Ignoring malparsed backtrace: 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2.0 scrub ok , 2021-03-11T17:29:26.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: cluster , 2021-03-11T17:29:26.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:25 smithi098 conmon[25960]: 2021-03-11T17:29:24.577623+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:26.989 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: 03-11T17:29:25.704899+0000 mon.a (mon.0) 2293 , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in , 2021-03-11T17:29:26.990 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:26 smithi059 conmon[25665]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:29.044 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: cephadm , 2021-03-11T17:29:29.045 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:28 smithi098 conmon[25960]: 2021-03-11T17:29:28 Ignoring malparsed backtrace: 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster 2021-03-11T17:29:30.961183+0000 osd.0 (osd.0 , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: ) 216 : cluster [DBG] 6.16 scrub ok , 2021-03-11T17:29:32.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:31 smithi098 conmon[25960]: cluster Ignoring malparsed backtrace: 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: 088393+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.d deep-scrub ok , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: cluster 2021-03-11T17:29 , 2021-03-11T17:29:35.118 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:34 smithi059 conmon[31137]: :34.605509+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1767: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 575 B/s rd, 0 op/s Ignoring malparsed backtrace: 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: [DBG] 4.6 scrub ok , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:39.370 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:38 smithi059 conmon[31137]: 2021-03-11T17:29:38.197369+0000 osd.1 (osd.1) 34 : Ignoring malparsed backtrace: 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: :42.098597+0000 osd.3 (osd.3) 269 : cluster [DBG] 7.2 scrub ok , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: , 2021-03-11T17:29:43.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:42 smithi098 conmon[25960]: cluster 2021- Ignoring malparsed backtrace: 2021-03-11T17:29:46.126 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:45 smithi059 conmon[25665]: mon.a (mon.0) 2303 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in , 2021-03-11T17:29:46.292 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: cluster 2021-03-11T17:29:44.889592 , 2021-03-11T17:29:46.293 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:45 smithi098 conmon[25960]: +0000 osd.0 (osd.0) 220 : cluster [DBG] 3.23 deep-scrub ok Ignoring malparsed backtrace: 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: 7) 147 : cluster [DBG] purged_snaps scrub starts , 2021-03-11T17:29:52.046 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: cluster 2021-03 , 2021-03-11T17:29:52.049 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 11 17:29:51 smithi098 conmon[25960]: -11T17:29:50.704353+0000 Ignoring malparsed backtrace: 2021-03-11T17:29:54.066 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: [DBG] 3.3 scrub ok , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: cluster 2021-03-11T17: , 2021-03-11T17:29:54.067 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:53 smithi059 conmon[31137]: 29:52.695386+0000 osd.5 (osd.5) Ignoring malparsed backtrace: 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: cluster , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: 2021-03-11T17:29:55.907185+0000 , 2021-03-11T17:29:57.116 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:29:56 smithi059 conmon[31137]: mon.a (mon.0) 2308 : cluster [DBG] Ignoring malparsed backtrace: 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 03-11T17:29:58.618225+0000 mgr.y (mgr. , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: 14144) 1080 : cluster [DBG] , 2021-03-11T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:29:59 smithi059 conmon[25665]: pgmap v1783: 168 pgs: 168 active+clean; 759 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 19 op/s Ignoring malparsed backtrace: 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: 2021-03-11T17:30:00.635491+0000 osd.5 ( , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: osd.5) 174 : cluster [DBG] 5.16 scrub ok , 2021-03-11T17:30:02.119 INFO:journalctl@ceph.mon.c.smithi059.stdout:Mar 11 17:30:01 smithi059 conmon[31137]: cluster Ignoring malparsed backtrace: 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 17:30:04.603746+0000 osd.5 (osd. , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: 5) 179 : cluster [DBG] 5.7 scrub ok , 2021-03-11T17:30:05.868 INFO:journalctl@ceph.mon.a.smithi059.stdout:Mar 11 17:30:05 smithi059 conmon[25665]: cluster 2021-03-11T17:30:04. Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' , 2021-03-11T17:30:07.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring Ignoring malparsed backtrace: 2021-03-11T17:30:10.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@mon.c.service' , 2021-03-11T17:30:10.151 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:10.152 INFO:tasks.cephadm.mon.c:Stopped mon.c Ignoring malparsed backtrace: 2021-03-11T17:30:25.117 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:24 smithi059 podman[194960]: 2021-03-11 17:30:24.975894615 +0000 UTC m=+0.377059428 container attach c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-swagner-testing-2021-03-11-1320-88e3ea0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4c4b735e251847c21869414e9d5a843ef1e7215f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) , 2021-03-11T17:30:25.617 INFO:journalctl@ceph.osd.3.smithi059.stdout:Mar 11 17:30:25 smithi059 podman[194960]: 2021-03-11 17:30:25.336858583 +0000 UTC m=+0.738023439 container died c1ad8eb4f2f164a08b9b8014e27e6ef9bcc14e27277e4fda90ad1f34934e6c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:631f808355e0d8b72c876d35e1b713c243f68053f97f3f432fa6ea04bea94965, name=ceph-946db372-828b-11eb-9083-001a4aab830c-osd.3-deactivate) , 2021-03-11T17:30:26.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@osd.3.service' Ignoring malparsed backtrace: 2021-03-11T17:30:50.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-946db372-828b-11eb-9083-001a4aab830c@prometheus.a.service' , 2021-03-11T17:30:50.385 DEBUG:teuthology.orchestra.run:got remote process result: None , 2021-03-11T17:30:50.386 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 479, in wait , proc.wait() Found 5 distinct failure reasons Failure: Command failed on smithi186 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2de454c-8285-11eb-9082-001a4aab830c -- ceph orch apply iscsi iscsi user password --placement '1;smithi186=iscsi.a'" 5 jobs: ['5956097', '5956129', '5956127', '5956144', '5956113'] suites intersection: [] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados:cephadm/smoke-roleless/{distro/ubuntu_20.04_kubic_stable', 'rados:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04', 'start}'] Failure: Command failed on smithi135 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:88e3ea0854f69cc01782866be7a1c31166ea1c79 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1d9d8ae-8288-11eb-9083-001a4aab830c -- ceph orch host ls --format=json' 4 jobs: ['5956134', '5956128', '5956123', '5956112'] suites intersection: ['fixed-2'] suites union: ['1-start', '2-thrash', '3-tasks/rados_api_tests', '3-tasks/small-objects', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v2only', 'rados:cephadm/thrash/{0-distro/ubuntu_20.04_kubic_stable', 'rados:cephadm/with-work/{0-distro/ubuntu_20.04_kubic_stable', 'rados:cephadm/with-work/{0-distro/ubuntu_20.04_kubic_testing', 'root}', 'start', 'tasks/rados_api_tests}', 'tasks/rados_python}'] Failure: Command failed (workunit test cephadm/create_iscsi_disks.sh) on smithi197 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=88e3ea0854f69cc01782866be7a1c31166ea1c79 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' 2 jobs: ['5956092', '5956125'] suites intersection: ['task/test_e2e}'] suites union: ['rados:cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable', 'rados:cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_e2e}'] Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) 1 jobs: ['5956133'] suites: ['mon_election/connectivity', 'rados:cephadm/workunits/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_orch_cli}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 1 jobs: ['5956151'] suites: ['1-start', '2-thrash', '3-tasks/snaps-few-objects', 'fixed-2', 'msgr/async-v2only', 'rados:cephadm/thrash/{0-distro/ubuntu_20.04_kubic_testing', 'root}']