2020-12-16T15:59:57.242 INFO:root:teuthology version: 1.0.0-8e55e2ff 2020-12-16T15:59:57.244 INFO:teuthology.results:subset: '945/9999' 2020-12-16T15:59:57.244 INFO:teuthology.results:seed: '2477' 2020-12-16T22:33:04.992 INFO:root:teuthology version: 1.0.0-0e05d594 2020-12-16T22:33:05.004 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-17T09:49:31.390 INFO:teuthology.results:Tests finished! gathering results... 2020-12-17T09:50:14.467 INFO:scrape:Found 446 jobs 2020-12-17T09:50:15.591 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:15.592 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:15.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:15.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.594 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:15.594 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:15.595 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:15.596 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.596 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.597 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:15.597 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:15.598 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:15.599 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:15.599 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:15.600 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:15.601 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.601 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.602 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.602 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:15.603 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:15.604 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:15.604 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:15.605 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:15.606 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:15.606 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:15.607 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:15.607 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:15.608 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:15.609 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:15.609 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:15.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:15.611 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:15.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:15.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:15.613 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:15.614 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:15.614 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:15.615 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:15.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:15.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:15.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:15.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:15.619 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:15.619 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:15.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:15.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:15.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:15.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:15.624 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:15.624 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:15.734 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:15.735 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:15.735 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:15.736 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.736 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:15.737 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:15.738 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:15.738 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.739 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.739 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:15.740 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:15.741 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:15.741 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:15.742 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:15.742 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:15.743 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.744 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.744 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.745 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:15.746 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:15.746 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:15.747 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:15.747 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:15.748 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:15.749 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:15.749 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:15.750 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:15.750 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:15.751 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:15.751 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:15.752 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.753 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:15.753 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:15.754 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:15.755 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:15.755 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:15.756 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:15.757 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:15.757 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:15.758 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:15.759 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:15.759 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:15.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:15.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:15.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:15.763 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:15.764 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:15.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:15.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:15.766 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:15.767 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:15.874 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:15.875 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:15.876 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:15.877 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.877 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:15.878 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:15.878 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:15.879 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.880 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.880 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:15.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:15.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:15.882 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:15.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:15.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:15.884 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.884 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.885 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:15.886 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:15.886 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:15.887 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:15.887 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:15.888 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:15.889 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:15.889 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:15.890 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:15.890 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:15.891 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:15.891 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:15.892 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:15.893 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:15.893 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:15.894 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:15.895 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:15.895 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:15.896 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:15.896 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:15.897 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:15.898 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:15.899 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:15.899 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:15.900 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:15.900 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:15.901 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:15.902 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:15.903 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:15.904 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:15.905 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:15.905 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:15.906 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:15.907 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.015 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.016 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.017 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.017 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.018 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.019 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.019 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.020 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.020 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.021 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.022 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.022 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.023 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.023 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.024 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.025 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.025 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.026 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.027 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.027 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.028 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.028 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.029 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.030 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.030 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.031 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.031 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.032 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.034 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.035 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.035 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.036 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.036 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.037 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.038 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.038 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.039 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.040 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.043 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.043 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.045 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.046 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.046 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.047 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.048 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.048 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.158 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.159 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.160 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.161 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.161 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.162 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.162 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.163 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.163 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.164 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.165 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.165 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.166 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.166 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.167 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.168 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.168 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.169 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.169 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.170 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.171 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.171 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.172 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.172 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.173 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.173 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.174 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.175 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.175 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.176 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.176 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.177 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.178 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.178 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.179 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.180 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.180 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.181 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.182 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.182 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.183 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.184 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.184 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.185 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.186 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.188 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.189 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.189 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.190 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.191 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.191 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.297 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.298 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.301 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.304 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.307 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.308 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.310 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.311 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.311 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.313 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.314 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.315 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.315 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.316 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.317 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.317 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.318 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.319 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.319 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.320 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.321 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.322 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.323 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.323 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.324 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.325 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.326 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.327 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.328 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.329 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.330 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.435 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.436 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.437 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.438 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.439 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.439 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.440 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.441 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.442 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.442 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.443 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.443 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.444 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.445 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.445 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.446 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.447 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.447 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.448 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.448 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.450 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.450 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.451 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.451 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.452 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.453 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.453 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.454 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.454 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.455 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.456 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.456 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.457 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.458 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.458 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.459 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.460 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.460 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.461 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.462 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.462 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.463 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.465 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.465 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.466 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.467 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.467 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.468 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.591 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.592 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.594 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.594 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.595 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.596 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.596 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.597 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.597 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.598 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.599 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.599 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.600 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.601 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.601 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.602 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.602 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.603 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.603 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.604 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.605 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.605 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.606 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.607 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.607 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.608 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.608 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.609 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.611 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.613 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.614 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.614 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.615 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.619 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.624 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.730 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.731 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.731 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.732 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.733 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.733 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.734 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.735 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.735 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.736 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.736 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.737 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.737 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.738 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.739 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.739 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.740 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.741 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.741 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.742 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.742 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.743 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.744 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.744 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.745 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.745 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.746 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.747 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.747 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.748 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.748 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.749 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.750 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.750 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.751 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.752 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.752 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.753 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.754 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.755 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.755 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.756 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.756 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.757 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.758 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.763 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:16.878 WARNING:scrape:Ignoring malparsed backtrace: ceph version 13.2.10-53-gc71791a171 (c71791a17113e678409221875b88b434b04e94a8) mimic (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f6620a9e96e] , 2: (()+0x287af7) [0x7f6620a9eaf7] 2020-12-17T09:50:16.879 WARNING:scrape:Ignoring malparsed backtrace: noscrub is unset , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.261369+0000 mgr.x (mgr.24118) 567 : cluster [DBG] pgmap v1137: 49 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 47 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 13 MiB/s wr, 33 op/s , 2020-12-16T19:28:47.863 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:28:47 smithi035 bash[12923]: cluster 2020-12-16T19:28:46.866306+0000 osd.11 (osd.11) 43 : cluster [DBG] 1.b scrub ok 2020-12-17T09:50:16.880 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:48.039 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.634368+0000", , 2020-12-16T19:28:48.040 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 5.9009999999999999e-06 2020-12-17T09:50:16.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.060 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.061 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.881 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.081 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:48.082 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:41.869803+0000", 2020-12-17T09:50:16.882 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.102 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.103 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.882 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: ] , 2020-12-16T19:28:48.123 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.883 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.144 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 8.1258e-05 , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.145 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.884 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:23.084511+0000", , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00022430900000000001 , 2020-12-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.884 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:24:57.172161+0000", , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.83171324099999999 , 2020-12-16T19:28:48.187 INFO:teuthology.orchestra.run.smithi035.stdout: } 2020-12-17T09:50:16.885 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:27:42.260860+0000", , 2020-12-16T19:28:48.208 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.1862999999999998e-05 2020-12-17T09:50:16.885 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", , 2020-12-16T19:28:48.229 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.558572+0000", 2020-12-17T09:50:16.886 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.250 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:48.251 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:28:28.847238+0000", 2020-12-17T09:50:16.887 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:48.271 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:48.272 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "queued_for_pg", 2020-12-17T09:50:16.887 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.206665+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.6 scrub ok , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.264067+0000 mgr.x (mgr.24118) 570 : cluster [DBG] pgmap v1143: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 820 KiB/s wr, 1 op/s , 2020-12-16T19:28:53.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:28:53 smithi049 bash[13001]: cluster 2020-12-16T19:28:52.571252+0000 osd.8 (osd.8) 35 : cluster [DBG] 1.17 scrub ok 2020-12-17T09:50:16.888 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:43.434031+0000", , 2020-12-16T19:28:58.593 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.000169476 , 2020-12-16T19:28:58.594 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.888 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:44.212318+0000", , 2020-12-16T19:28:58.614 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.00885968 , 2020-12-16T19:28:58.615 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.889 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:19:48.059929+0000", , 2020-12-16T19:28:58.636 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 7.6406e-05 , 2020-12-16T19:28:58.637 INFO:teuthology.orchestra.run.smithi035.stdout: }, 2020-12-17T09:50:16.890 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.658 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.97253395799999998, , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { , 2020-12-16T19:28:58.659 INFO:teuthology.orchestra.run.smithi035.stdout: "flag_point": "commit sent; apply or cleanup", 2020-12-17T09:50:16.890 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:20:27.274903+0000", , 2020-12-16T19:28:58.680 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 9.0089999999999996e-06 2020-12-17T09:50:16.891 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "done", , 2020-12-16T19:28:58.700 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:21:24.678167+0000", , 2020-12-16T19:28:58.701 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.55342259500000002 2020-12-17T09:50:16.891 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "age": 416.45999315400002, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 0.71387725999999996, , 2020-12-16T19:28:58.721 INFO:teuthology.orchestra.run.smithi035.stdout: "type_data": { 2020-12-17T09:50:16.892 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.741 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247130+0000", 2020-12-17T09:50:16.892 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.761 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "waiting for blocked object", , 2020-12-16T19:28:58.762 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.247336+0000", 2020-12-17T09:50:16.893 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.781 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", , 2020-12-16T19:28:58.782 INFO:teuthology.orchestra.run.smithi035.stdout: "time": "2020-12-16T19:22:33.768232+0000", 2020-12-17T09:50:16.894 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.801 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.802 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "throttled", 2020-12-17T09:50:16.894 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.822 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.823 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "header_read", 2020-12-17T09:50:16.895 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.842 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.843 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "all_read", 2020-12-17T09:50:16.895 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.863 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "dispatched", 2020-12-17T09:50:16.896 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: { , 2020-12-16T19:28:58.883 INFO:teuthology.orchestra.run.smithi035.stdout: "event": "reached_pg", 2020-12-17T09:50:16.897 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: "duration": 4294967295.9999681 , 2020-12-16T19:28:58.909 INFO:teuthology.orchestra.run.smithi035.stdout: }, , 2020-12-16T19:28:58.910 INFO:teuthology.orchestra.run.smithi035.stdout: { 2020-12-17T09:50:16.897 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.715479+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in , 2020-12-16T19:29:05.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:04 smithi035 bash[12923]: cluster 2020-12-16T19:29:03.949417+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.19 deep-scrub ok , 2020-12-16T19:29:05.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:04 smithi186 bash[12658]: audit 2020-12-16T19:29:03.715173+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? v1:172.21.15.49:0/1049025504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2020-12-17T09:50:16.898 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.2 to 1 (10000) , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: cluster 2020-12-16T19:29:20.275905+0000 mgr.x (mgr.24118) 584 : cluster [DBG] pgmap v1168: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:21.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:21 smithi035 bash[12923]: audit 2020-12-16T19:29:20.456544+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? v1:172.21.15.49:0/1625742340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2020-12-17T09:50:16.899 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:34.638 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:29:34 smithi186 bash[12658]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:34 smithi049 bash[13001]: audit 2020-12-16T19:29:33.291126+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.49:0/277482206' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2020-12-16T19:29:34.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd reweight 4 1 2020-12-17T09:50:16.899 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:29:47 smithi049 bash[13001]: cluster 2020-12-16T19:29:46.631546+0000 osd.0 (osd.0) 46 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:29:47.869 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: cluster 2020-12-16T19:29:46.287294+0000 mgr.x (mgr.24118) 597 : cluster [DBG] pgmap v1191: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:29:47.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:47 smithi035 bash[12923]: audit 2020-12-16T19:29:46.376038+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.49:0/279906322' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2020-12-17T09:50:16.900 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:29:59.620 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: audit 2020-12-16T19:29:58.340788+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.49:0/1227002085' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.340947+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in , 2020-12-16T19:29:59.621 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:29:59 smithi035 bash[12923]: cluster 2020-12-16T19:29:58.371076+0000 osd.1 (osd.1) 53 : cluster [DBG] 1.3 scrub ok 2020-12-17T09:50:16.900 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: cluster 2020-12-16T19:30:06.295951+0000 mgr.x (mgr.24118) 607 : cluster [DBG] pgmap v1210: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:07.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:07 smithi049 bash[13001]: audit 2020-12-16T19:30:06.405465+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? v1:172.21.15.49:0/1219533492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2020-12-17T09:50:16.901 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: audit 2020-12-16T19:30:18.473280+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? v1:172.21.15.49:0/3252012196' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished , 2020-12-16T19:30:19.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.473489+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in , 2020-12-16T19:30:19.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:19 smithi035 bash[12923]: cluster 2020-12-16T19:30:18.673127+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub starts 2020-12-17T09:50:16.902 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:29.887748+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.19 scrub ok , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: cluster 2020-12-16T19:30:30.306610+0000 mgr.x (mgr.24118) 619 : cluster [DBG] pgmap v1233: 49 pgs: 1 active+clean+scrubbing, 48 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:30:31 smithi035 bash[12923]: audit 2020-12-16T19:30:30.555126+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.49:0/2016929440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2020-12-17T09:50:16.903 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:30:47.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.313078+0000 mgr.x (mgr.24118) 627 : cluster [DBG] pgmap v1245: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.380651+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in , 2020-12-16T19:30:47.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:30:47 smithi049 bash[13001]: cluster 2020-12-16T19:30:46.403353+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.5 scrub ok 2020-12-17T09:50:16.903 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:03.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.225290+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.227831+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:03.871 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:03 smithi035 bash[12923]: cluster 2020-12-16T19:31:02.319992+0000 mgr.x (mgr.24118) 635 : cluster [DBG] pgmap v1257: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.904 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.709183+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:19.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:19 smithi049 bash[13001]: cluster 2020-12-16T19:31:18.723310+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:20.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:20 smithi049 bash[13001]: cluster 2020-12-16T19:31:19.540527+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2020-12-17T09:50:16.904 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:31.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:37517df36b82342cd0353231eb55b40ee1f29ef4 shell --fsid 1f481d7c-3fd2-11eb-9813-001a4aab830c -- ceph osd last-stat-seq osd.1 , Inferring config /var/lib/ceph/1f481d7c-3fd2-11eb-9813-001a4aab830c/mon.a/config , 2020-12-16T19:31:31.870 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:31 smithi035 bash[12923]: cluster 2020-12-16T19:31:30.333119+0000 mgr.x (mgr.24118) 652 : cluster [DBG] pgmap v1277: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-17T09:50:16.905 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:43.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok , 2020-12-16T19:31:43.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:43 smithi049 bash[13001]: cluster 2020-12-16T19:31:42.338577+0000 mgr.x (mgr.24118) 658 : cluster [DBG] pgmap v1285: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:31:44.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:31:43 smithi035 bash[12923]: cluster 2020-12-16T19:31:41.990318+0000 osd.1 (osd.1) 63 : cluster [DBG] 1.16 deep-scrub ok 2020-12-17T09:50:16.906 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:31:52.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.700129+0000 osd.10 (osd.10) 76 : cluster [DBG] purged_snaps scrub starts , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:51.708109+0000 osd.10 (osd.10) 77 : cluster [DBG] purged_snaps scrub ok , 2020-12-16T19:31:52.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:31:52 smithi049 bash[13001]: cluster 2020-12-16T19:31:52.174964+0000 osd.7 (osd.7) 86 : cluster [DBG] 2.0 scrub ok 2020-12-17T09:50:16.908 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:08.120 INFO:journalctl@ceph.mon.c.smithi035.stdout:Dec 16 19:32:07 smithi035 bash[12923]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: cluster 2020-12-16T19:32:06.349438+0000 mgr.x (mgr.24118) 670 : cluster [DBG] pgmap v1301: 49 pgs: 49 active+clean; 2.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-16T19:32:08.138 INFO:journalctl@ceph.mon.b.smithi186.stdout:Dec 16 19:32:07 smithi186 bash[12658]: audit 2020-12-16T19:32:07.105352+0000 mon.a (mon.0) 1923 : audit [DBG] from='client.? v1:172.21.15.49:0/1402833465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2020-12-17T09:50:16.908 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:21.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.446892+0000 osd.3 (osd.3) 58 : cluster [DBG] 1.c scrub ok , 2020-12-16T19:32:21.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:21 smithi049 bash[13001]: cluster 2020-12-16T19:32:20.769876+0000 osd.0 (osd.0) 58 : cluster [DBG] 1.1f scrub ok , 2020-12-16T19:32:22.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Dec 16 19:32:22 smithi049 bash[13001]: audit 2020-12-16T19:32:22.078400+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.49:0/3737476373' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2020-12-17T09:50:16.909 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:30 smithi049 docker[96374]: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1 , 2020-12-16T19:32:31.457 INFO:journalctl@ceph.osd.1.smithi049.stdout:Dec 16 19:32:31 smithi049 bash[96418]: Error: No such container: ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c-osd.1-deactivate , 2020-12-16T19:32:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.1.service' 2020-12-17T09:50:16.909 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:32:54.341 INFO:journalctl@ceph.osd.10.smithi035.stdout:Dec 16 19:32:54 smithi035 bash[19947]: debug 2020-12-16T19:32:54.037+0000 7f9ef263a700 -1 osd.10 731 heartbeat_check: no reply from 172.21.15.49:6820 osd.2 since back 2020-12-16T19:32:29.196264+0000 front 2020-12-16T19:32:29.196547+0000 (oldest deadline 2020-12-16T19:32:53.896456+0000) , 2020-12-16T19:32:54.816 DEBUG:teuthology.orchestra.run.smithi035:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1f481d7c-3fd2-11eb-9813-001a4aab830c@osd.9.service' , 2020-12-16T19:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-12-17T09:50:16.910 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-16T19:33:25.516 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2020-12-16T19:33:25.528 INFO:teuthology.task.clock:Checking final clock skew... , 2020-12-16T19:33:25.528 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-12-17T09:50:16.911 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2020-12-17T09:50:20.023 INFO:scrape:Found 11 distinct failure reasons 2020-12-17T09:50:20.025 INFO:scrape:Dead: b'2020-12-17T07:40:39.826 DEBUG:teuthology.exit:Finished running handlers' 2020-12-17T09:50:20.025 INFO:scrape:16 jobs: ['5713463', '5713413', '5713538', '5713619', '5713356', '5713486', '5713433', '5713385', '5713512', '5713639', '5713530', '5713255', '5713564', '5713335', '5713301', '5713349'] 2020-12-17T09:50:20.026 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados'] 2020-12-17T09:50:20.027 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'thrashosds-health}', 'validater/valgrind}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache}', 'workloads/ec-small-objects-many-deletes}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_set_object}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] 2020-12-17T09:50:20.027 INFO:scrape: 2020-12-17T09:50:20.027 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2020-12-17T09:50:20.028 INFO:scrape:3 jobs: ['5713472', '5713267', '5713358'] 2020-12-17T09:50:20.028 INFO:scrape:suites intersection: ['mon_election/classic', 'openstack}', 'rados', 'thrashosds-health'] 2020-12-17T09:50:20.029 INFO:scrape:suites union: ['ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-snaps-few-objects-overwrites}'] 2020-12-17T09:50:20.029 INFO:scrape: 2020-12-17T09:50:20.030 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=32567) 2020-12-17T09:50:20.030 INFO:scrape:1 jobs: ['5713309'] 2020-12-17T09:50:20.031 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-big}'] 2020-12-17T09:50:20.031 INFO:scrape: 2020-12-17T09:50:20.031 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-12-17T09:50:20.032 INFO:scrape:6 jobs: ['5713467', '5713521', '5713351', '5713623', '5713287', '5713399'] 2020-12-17T09:50:20.032 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-12-17T09:50:20.033 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-12-17T09:50:20.033 INFO:scrape: 2020-12-17T09:50:20.034 INFO:scrape:Failure: "2020-12-16T16:45:07.167752+0000 osd.0 (osd.0) 165 : cluster [ERR] 58.5 shard 0 soid 58:a16d1d95:test-rados-api-smithi092-23179-74::foo:23 : candidate size 10 info size 0 mismatch" in cluster log 2020-12-17T09:50:20.034 INFO:scrape:2 jobs: ['5713250', '5713632'] 2020-12-17T09:50:20.034 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-12-17T09:50:20.035 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-12-17T09:50:20.035 INFO:scrape: 2020-12-17T09:50:20.036 INFO:scrape:Failure: Command failed (workunit test rados/test_python.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' 2020-12-17T09:50:20.036 INFO:scrape:1 jobs: ['5713555'] 2020-12-17T09:50:20.037 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_python}'] 2020-12-17T09:50:20.037 INFO:scrape: 2020-12-17T09:50:20.038 INFO:scrape:Failure: "2020-12-16T22:07:33.717152+0000 mon.a (mon.0) 1181 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log 2020-12-17T09:50:20.038 INFO:scrape:1 jobs: ['5713636'] 2020-12-17T09:50:20.038 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2020-12-17T09:50:20.039 INFO:scrape: 2020-12-17T09:50:20.039 INFO:scrape:Assertion: /build/ceph-13.2.10-53-gc71791a1/src/test/osd/RadosModel.h: 1829: FAILED assert((int)(info->id) > last_finished) 2020-12-17T09:50:20.040 INFO:scrape:1 jobs: ['5713403'] 2020-12-17T09:50:20.040 INFO:scrape:suites: ['1-install/mimic', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}'] 2020-12-17T09:50:20.041 INFO:scrape: 2020-12-17T09:50:20.041 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-12-17T09:50:20.041 INFO:scrape:3 jobs: ['5713505', '5713669', '5713336'] 2020-12-17T09:50:20.042 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2020-12-17T09:50:20.042 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-17T09:50:20.043 INFO:scrape: 2020-12-17T09:50:20.043 INFO:scrape:Failure: Command failed (workunit test osd/osd-backfill-stats.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh' 2020-12-17T09:50:20.044 INFO:scrape:1 jobs: ['5713605'] 2020-12-17T09:50:20.044 INFO:scrape:suites: ['rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{rhel_8}', 'workloads/osd}'] 2020-12-17T09:50:20.044 INFO:scrape: 2020-12-17T09:50:20.045 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-12-17T09:50:20.045 INFO:scrape:1 jobs: ['5713291'] 2020-12-17T09:50:20.046 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-12-17T09:50:20.046 INFO:scrape: 2020-12-17T09:50:20.061 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/ failed: 19 dead: 17 running: 0 waiting: 0 queued: 0 passed: 410 Fail ================================================================= [5713250] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:26:03 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713250/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713250/ "2020-12-16T16:45:07.167752+0000 osd.0 (osd.0) 165 : cluster [ERR] 58.5 shard 0 soid 58:a16d1d95:test-rados-api-smithi092-23179-74::foo:23 : candidate size 10 info size 0 mismatch" in cluster log [5713267] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 01:51:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713267/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713267/ Command failed on smithi162 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5713287] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:17:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713287/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06801f08d0ee4d4eb3370b824daa6c47 reached maximum tries (180) after waiting for 180 seconds [5713291] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:08:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713291/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713291/ sentry: https://sentry.ceph.com/organizations/ceph/?query=865433ae773f4a8fbe94b57c298e7b64 Command failed (workunit test cephadm/test_adoption.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5713336] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:10:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713336/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cae5d03c4c0b442599297509f6d23ec8 Command failed (workunit test cephadm/test_cephadm.sh) on smithi144 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5713351] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:16:25 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713351/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7afa3f13e0448bf840e66b98a740c79 reached maximum tries (180) after waiting for 180 seconds [5713358] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:45:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713358/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713358/ Command failed on smithi143 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5713399] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713399/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e096f3299e1e4855aa2cfb0eea30f037 reached maximum tries (180) after waiting for 180 seconds [5713403] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:28:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713403/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713403/ Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5713467] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713467/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713467/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac960e4d2cf6496e89de2b3135e570bd reached maximum tries (180) after waiting for 180 seconds [5713472] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 01:15:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713472/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713472/ Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5713505] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:10:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713505/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae7fcc4e338248a69425f9eae368501f Command failed (workunit test cephadm/test_cephadm.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5713521] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713521/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713521/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afe164f4b0204f21be5e0449855ecea0 reached maximum tries (180) after waiting for 180 seconds [5713555] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} ----------------------------------------------------------------- time: 00:12:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713555/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713555/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d3a09bab45242c6a77e02d7a1087694 Command failed (workunit test rados/test_python.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' [5713605] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd} ----------------------------------------------------------------- time: 01:38:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713605/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713605/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d52a2fd9eb54f67a86e159f60b7f22d Command failed (workunit test osd/osd-backfill-stats.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill- stats.sh' [5713623] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:59 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713623/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec678a02189e49a4891f3b81410b1d36 reached maximum tries (180) after waiting for 180 seconds [5713632] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:36:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713632/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713632/ "2020-12-16T22:11:36.020774+0000 osd.6 (osd.6) 70 : cluster [ERR] 56.2 shard 0 soid 56:46dcae26:test-rados-api-smithi190-41309-74::foo:23 : candidate size 10 info size 0 mismatch" in cluster log [5713636] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 01:10:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713636/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713636/ "2020-12-16T22:07:33.717152+0000 mon.a (mon.0) 1181 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log [5713669] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:08:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713669/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39492d63379a4d31bc1ee42aa0059508 Command failed (workunit test cephadm/test_cephadm.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37517df36b82342cd0353231eb55b40ee1f29ef4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' Dead ================================================================= [5713255] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713255/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713255/ [5713301] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713301/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713301/ [5713309] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 11:48:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713309/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713309/ psutil.NoSuchProcess process no longer exists (pid=32567) [5713335] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713335/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713335/ [5713349] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713349/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713349/ [5713356] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713356/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713356/ [5713385] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713385/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713385/ [5713413] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713413/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713413/ [5713433] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713433/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713433/ [5713463] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713463/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713463/ [5713486] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713486/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713486/ [5713512] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713512/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713512/ [5713530] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713530/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713530/ [5713538] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713538/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713538/ [5713564] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713564/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713564/ [5713619] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713619/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713619/ [5713639] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713639/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713639/ Pass ================================================================= [5713238] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713238/ [5713239] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:14:19 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713239/ [5713240] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_5925} time: 00:10:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713240/ [5713241] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:15:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713241/ [5713242] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:25:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713242/ [5713243] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache} time: 00:14:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713243/ [5713244] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:12:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713244/ [5713245] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:22:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713245/ [5713246] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713246/ [5713247] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} time: 00:08:31 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713247/ [5713248] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:28 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713248/ [5713249] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:18:59 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713249/ [5713251] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713251/ [5713252] rados/singleton/{all/mon-config mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:10:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713252/ [5713253] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:18:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713253/ [5713254] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:38:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713254/ [5713256] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/failover} time: 00:13:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713256/ [5713257] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713257/ [5713258] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/radosbench} time: 02:06:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713258/ [5713259] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:32:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713259/ [5713260] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713260/ [5713261] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:13:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713261/ [5713262] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:31:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713262/ [5713263] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:19:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713263/ [5713264] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:10:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713264/ [5713265] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:12:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713265/ [5713266] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:34:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713266/ [5713268] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/mon_recovery validater/valgrind} time: 00:29:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713268/ [5713269] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:13:28 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713269/ [5713270] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:17:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713270/ [5713271] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713271/ [5713272] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713272/ [5713273] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:14:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713273/ [5713274] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713274/ [5713275] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:20:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713275/ [5713276] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:05:34 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713276/ [5713277] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:21:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713277/ [5713278] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713278/ [5713279] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:19:32 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713279/ [5713280] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:20:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713280/ [5713281] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:23:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713281/ [5713282] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-radosbench} time: 02:32:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713282/ [5713283] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713283/ [5713284] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:08:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713284/ [5713285] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:24:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713285/ [5713286] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:08:34 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713286/ [5713288] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:13:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713288/ [5713289] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713289/ [5713290] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:57 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713290/ [5713292] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713292/ [5713293] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:24:34 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713293/ [5713294] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713294/ [5713295] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:21:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713295/ [5713296] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713296/ [5713297] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{rhel_8}} time: 00:13:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713297/ [5713298] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:20:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713298/ [5713299] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:24:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713299/ [5713300] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/insights} time: 00:13:03 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713300/ [5713302] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:57 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713302/ [5713303] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:05:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713303/ [5713304] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:32:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713304/ [5713305] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:12:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713305/ [5713306] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:59 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713306/ [5713307] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:30:19 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713307/ [5713308] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:25 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713308/ [5713310] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713310/ [5713311] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/lockdep} time: 00:11:59 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713311/ [5713312] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713312/ [5713313] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713313/ [5713314] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:14:12 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713314/ [5713315] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:13:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713315/ [5713316] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:25:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713316/ [5713317] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713317/ [5713318] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:19:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713318/ [5713319] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench} time: 02:27:03 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713319/ [5713320] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713320/ [5713321] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:08:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713321/ [5713322] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713322/ [5713323] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:24:21 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713323/ [5713324] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:33:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713324/ [5713325] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713325/ [5713326] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:58:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713326/ [5713327] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:11:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713327/ [5713328] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced} time: 00:22:57 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713328/ [5713329] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:25:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713329/ [5713330] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713330/ [5713331] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:21:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713331/ [5713332] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:33:13 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713332/ [5713333] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:29:19 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713333/ [5713334] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713334/ [5713337] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:09:11 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713337/ [5713338] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713338/ [5713339] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713339/ [5713340] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:32:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713340/ [5713341] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:28:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713341/ [5713342] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:20:11 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713342/ [5713343] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:21:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713343/ [5713344] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713344/ [5713345] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:27:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713345/ [5713346] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:14:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713346/ [5713347] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:24:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713347/ [5713348] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:43:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713348/ [5713350] rados/singleton/{all/radostool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:11:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713350/ [5713352] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:12 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713352/ [5713353] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench} time: 01:56:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713353/ [5713354] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:21:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713354/ [5713355] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:11 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713355/ [5713357] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect} time: 00:15:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713357/ [5713359] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713359/ [5713360] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713360/ [5713361] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:15:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713361/ [5713362] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:21:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713362/ [5713363] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713363/ [5713364] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:14:30 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713364/ [5713365] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:35 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713365/ [5713366] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:26:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713366/ [5713367] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713367/ [5713368] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} time: 00:30:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713368/ [5713369] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:19:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713369/ [5713370] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:14:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713370/ [5713371] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:13:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713371/ [5713372] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713372/ [5713373] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos} time: 00:04:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713373/ [5713374] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:19:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713374/ [5713375] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:16:21 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713375/ [5713376] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} time: 00:11:57 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713376/ [5713377] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713377/ [5713378] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:24:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713378/ [5713379] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:22:03 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713379/ [5713380] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/osd} time: 03:05:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713380/ [5713381] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects} time: 00:19:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713381/ [5713382] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713382/ [5713383] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713383/ [5713384] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/progress} time: 00:18:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713384/ [5713386] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:33:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713386/ [5713387] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:21:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713387/ [5713388] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:31:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713388/ [5713389] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713389/ [5713390] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713390/ [5713391] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:25:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713391/ [5713392] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713392/ [5713393] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_8}} time: 05:03:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713393/ [5713394] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:12:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713394/ [5713395] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713395/ [5713396] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713396/ [5713397] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:16:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713397/ [5713398] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:06:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713398/ [5713400] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713400/ [5713401] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:45:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713401/ [5713402] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713402/ [5713404] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:09:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713404/ [5713405] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-big} time: 00:20:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713405/ [5713406] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:16:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713406/ [5713407] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:13:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713407/ [5713408] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:11 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713408/ [5713409] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} time: 00:12:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713409/ [5713410] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/scrub} time: 01:39:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713410/ [5713411] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/snaps-few-objects} time: 00:27:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713411/ [5713412] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:10:25 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713412/ [5713414] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:33:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713414/ [5713415] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:24:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713415/ [5713416] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:29:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713416/ [5713417] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:18:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713417/ [5713418] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713418/ [5713419] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps} time: 00:20:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713419/ [5713420] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:34:51 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713420/ [5713421] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:43 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713421/ [5713422] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:24:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713422/ [5713423] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:13:43 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713423/ [5713424] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/prometheus} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713424/ [5713425] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713425/ [5713426] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713426/ [5713427] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:25:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713427/ [5713428] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713428/ [5713429] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:19:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713429/ [5713430] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:31:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713430/ [5713431] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:14:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713431/ [5713432] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:13:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713432/ [5713434] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 01:04:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713434/ [5713435] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:25:21 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713435/ [5713436] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:21:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713436/ [5713437] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713437/ [5713438] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713438/ [5713439] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} time: 00:30:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713439/ [5713440] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:09:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713440/ [5713441] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713441/ [5713442] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_8}} time: 00:09:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713442/ [5713443] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:07:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713443/ [5713444] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:32:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713444/ [5713445] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713445/ [5713446] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush} time: 00:11:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713446/ [5713447] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 05:17:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713447/ [5713448] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:31 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713448/ [5713449] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 01:01:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713449/ [5713450] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:21:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713450/ [5713451] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 02:14:53 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713451/ [5713452] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:32:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713452/ [5713453] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:20:51 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713453/ [5713454] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect} time: 00:12:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713454/ [5713455] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}} time: 00:13:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713455/ [5713456] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713456/ [5713457] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/pool-create-delete} time: 00:17:31 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713457/ [5713458] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} time: 00:20:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713458/ [5713459] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:15:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713459/ [5713460] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:13 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713460/ [5713461] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:14:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713461/ [5713462] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:21:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713462/ [5713464] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:20:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713464/ [5713465] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713465/ [5713466] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:17:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713466/ [5713468] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/workunits} time: 00:10:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713468/ [5713469] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:12 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713469/ [5713470] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:20:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713470/ [5713471] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713471/ [5713473] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:08:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713473/ [5713474] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:23:43 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713474/ [5713475] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:20:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713475/ [5713476] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:36:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713476/ [5713477] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:37:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713477/ [5713478] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:24:30 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713478/ [5713479] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713479/ [5713480] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:26:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713480/ [5713481] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:43 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713481/ [5713482] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:21 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713482/ [5713483] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:30 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713483/ [5713484] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713484/ [5713485] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:36:32 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713485/ [5713487] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:08:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713487/ [5713488] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713488/ [5713489] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713489/ [5713490] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:23:12 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713490/ [5713491] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:16:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713491/ [5713492] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:30:53 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713492/ [5713493] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:25 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713493/ [5713494] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:12:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713494/ [5713495] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:30 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713495/ [5713496] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:12:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713496/ [5713497] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713497/ [5713498] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_5925} time: 00:11:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713498/ [5713499] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:15:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713499/ [5713500] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:08:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713500/ [5713501] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:27:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713501/ [5713502] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:24:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713502/ [5713503] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:20:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713503/ [5713504] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:31:34 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713504/ [5713506] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:11:53 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713506/ [5713507] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:13:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713507/ [5713508] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr} time: 00:12:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713508/ [5713509] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713509/ [5713510] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:10:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713510/ [5713511] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/crash} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713511/ [5713513] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:22:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713513/ [5713514] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:12:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713514/ [5713515] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713515/ [5713516] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:24:19 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713516/ [5713517] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:25:35 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713517/ [5713518] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:17:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713518/ [5713519] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:07:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713519/ [5713520] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:25:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713520/ [5713522] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713522/ [5713523] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:07:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713523/ [5713524] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:20:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713524/ [5713525] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713525/ [5713526] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713526/ [5713527] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:12:37 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713527/ [5713528] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713528/ [5713529] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713529/ [5713531] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:26:28 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713531/ [5713532] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:39:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713532/ [5713533] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:07:32 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713533/ [5713534] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:23:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713534/ [5713535] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713535/ [5713536] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:53 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713536/ [5713537] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:25:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713537/ [5713539] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_api_tests} time: 00:32:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713539/ [5713540] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:31:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713540/ [5713541] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:08:49 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713541/ [5713542] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713542/ [5713543] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:32:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713543/ [5713544] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:28 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713544/ [5713545] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:05:51 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713545/ [5713546] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:52:22 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713546/ [5713547] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:07:29 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713547/ [5713548] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:06:30 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713548/ [5713549] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:26:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713549/ [5713550] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/redirect} time: 00:12:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713550/ [5713551] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713551/ [5713552] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:13:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713552/ [5713553] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:14:42 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713553/ [5713554] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:10:43 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713554/ [5713556] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:31:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713556/ [5713557] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:13:26 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713557/ [5713558] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:53 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713558/ [5713559] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713559/ [5713560] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:43:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713560/ [5713561] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:15:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713561/ [5713562] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/rbd_cls} time: 00:20:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713562/ [5713563] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:59 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713563/ [5713565] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:14:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713565/ [5713566] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713566/ [5713567] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713567/ [5713568] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:21:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713568/ [5713569] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713569/ [5713570] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/classic} time: 00:17:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713570/ [5713571] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mon} time: 00:42:21 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713571/ [5713572] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:08:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713572/ [5713573] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects} time: 00:16:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713573/ [5713574] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:22:21 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713574/ [5713575] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713575/ [5713576] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:11:41 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713576/ [5713577] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713577/ [5713578] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:12 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713578/ [5713579] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:36:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713579/ [5713580] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:12 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713580/ [5713581] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} time: 00:16:35 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713581/ [5713582] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_osdmap_prune} time: 00:58:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713582/ [5713583] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:21:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713583/ [5713584] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:33:36 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713584/ [5713585] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:31:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713585/ [5713586] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713586/ [5713587] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713587/ [5713588] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:39 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713588/ [5713589] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:17:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713589/ [5713590] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:57 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713590/ [5713591] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:17:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713591/ [5713592] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:35 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713592/ [5713593] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}} time: 02:44:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713593/ [5713594] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:19:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713594/ [5713595] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:36:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713595/ [5713596] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:10:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713596/ [5713597] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:54:24 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713597/ [5713598] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} time: 00:19:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713598/ [5713599] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:09:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713599/ [5713600] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-agent-big} time: 00:15:46 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713600/ [5713601] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713601/ [5713602] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:23:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713602/ [5713603] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-agent-small} time: 00:13:19 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713603/ [5713604] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713604/ [5713606] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:34:02 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713606/ [5713607] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:25:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713607/ [5713608] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713608/ [5713609] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:08:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713609/ [5713610] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:13:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713610/ [5713611] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} time: 00:21:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713611/ [5713612] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713612/ [5713613] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:12:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713613/ [5713614] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:16:11 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713614/ [5713615] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:23:31 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713615/ [5713616] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713616/ [5713617] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:25:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713617/ [5713618] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:45 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713618/ [5713620] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:30:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713620/ [5713621] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:12:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713621/ [5713622] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713622/ [5713624] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:11:33 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713624/ [5713625] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713625/ [5713626] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_mon_workunits} time: 00:25:20 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713626/ [5713627] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-radosbench} time: 01:45:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713627/ [5713628] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713628/ [5713629] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:08:54 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713629/ [5713630] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:07:23 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713630/ [5713631] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:10:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713631/ [5713633] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:22:06 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713633/ [5713634] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:13:17 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713634/ [5713635] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 01:37:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713635/ [5713637] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:13:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713637/ [5713638] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:29:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713638/ [5713640] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:26:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713640/ [5713641] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/radosbench} time: 01:03:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713641/ [5713642] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713642/ [5713643] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:06:50 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713643/ [5713644] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713644/ [5713645] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:10:10 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713645/ [5713646] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:58 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713646/ [5713647] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713647/ [5713648] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:12:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713648/ [5713649] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:07 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713649/ [5713650] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:44 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713650/ [5713651] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:15:48 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713651/ [5713652] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:10:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713652/ [5713653] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713653/ [5713654] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:20:55 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713654/ [5713655] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:17:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713655/ [5713656] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:11:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713656/ [5713657] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:36:14 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713657/ [5713658] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:21:27 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713658/ [5713659] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713659/ [5713660] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:15:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713660/ [5713661] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:14:11 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713661/ [5713662] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:22:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713662/ [5713663] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:32:30 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713663/ [5713664] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:10:04 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713664/ [5713665] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:31:25 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713665/ [5713666] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:47 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713666/ [5713667] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/crush} time: 00:12:08 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713667/ [5713668] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} time: 04:47:09 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713668/ [5713670] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:21:03 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713670/ [5713671] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/snaps-few-objects} time: 00:26:40 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713671/ [5713672] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:15:16 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713672/ [5713673] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} time: 00:24:18 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713673/ [5713674] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:18:01 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713674/ [5713675] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:20:52 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713675/ [5713676] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:26:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713676/ [5713677] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:09:32 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713677/ [5713678] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:43 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713678/ [5713679] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:13:05 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713679/ [5713680] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:10:15 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713680/ [5713681] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:07:28 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713681/ [5713682] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/progress} time: 00:24:38 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713682/ [5713683] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:26:53 info: http://pulpito.ceph.com/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713683/