2021-02-05T10:48:55.358 INFO:root:teuthology version: 1.0.0-5251f0da 2021-02-05T10:48:55.358 INFO:teuthology.results:subset: '9/100000' 2021-02-05T10:48:55.359 INFO:teuthology.results:seed: '3852' 2021-02-06T08:59:45.853 INFO:root:teuthology version: 1.1.0-17c09965 2021-02-06T08:59:48.073 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-02-06T13:06:57.046 INFO:teuthology.results:Tests finished! gathering results... 2021-02-06T13:08:19.029 INFO:scrape:Found 360 jobs 2021-02-06T13:08:28.664 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:28.664 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.664 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:28.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:28.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:28.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:28.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:28.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:28.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:28.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:28.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:28.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:28.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:28.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:28.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:28.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:28.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:28.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:28.668 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:28.669 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:28.669 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:28.669 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:28.669 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:28.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:28.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:28.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:28.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:28.745 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:28.745 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.746 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:28.746 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:28.747 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:28.747 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:28.748 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:28.748 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:28.748 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.748 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:28.748 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:28.749 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:28.749 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.749 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:28.749 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:28.750 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:28.750 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:28.750 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:28.750 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:28.751 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:28.751 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:28.751 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:28.751 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:28.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:28.752 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:28.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:28.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:28.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:28.753 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:28.815 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:28.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:28.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:28.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:28.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:28.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:28.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:28.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:28.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:28.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:28.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:28.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:28.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:28.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:28.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:28.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:28.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:28.820 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:28.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:28.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:28.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:28.821 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:28.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:28.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:28.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:28.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:28.889 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:28.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:28.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:28.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:28.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:28.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:28.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:28.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:28.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:28.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:28.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:28.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:28.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:28.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:28.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:28.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:28.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:28.895 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:28.895 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:28.895 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:28.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:28.896 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:28.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:28.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:28.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:28.897 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:28.979 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:28.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:28.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:28.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:28.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:28.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:28.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:28.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:28.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:28.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:28.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:28.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:28.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:28.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:28.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:28.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:28.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:28.984 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:28.984 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:28.984 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:28.984 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:28.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:28.985 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:28.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:28.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:28.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:28.986 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.074 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:29.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:29.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:29.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:29.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:29.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:29.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:29.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:29.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:29.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:29.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:29.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:29.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:29.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:29.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:29.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:29.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:29.079 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:29.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:29.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:29.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:29.080 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:29.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:29.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:29.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:29.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.174 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:29.175 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.175 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:29.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:29.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:29.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:29.177 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:29.177 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:29.178 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.178 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:29.179 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:29.179 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:29.180 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.180 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:29.181 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:29.181 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:29.182 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:29.182 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:29.183 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:29.183 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:29.185 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:29.186 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:29.186 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:29.187 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:29.187 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:29.188 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:29.188 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:29.189 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:29.189 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.256 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:29.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:29.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:29.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:29.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:29.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:29.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:29.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:29.259 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:29.259 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:29.259 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.259 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:29.260 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:29.260 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:29.260 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:29.260 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:29.261 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:29.261 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:29.261 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:29.261 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:29.261 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:29.262 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:29.262 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:29.262 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:29.262 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:29.263 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:29.263 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.335 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:29.335 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:29.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:29.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:29.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:29.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:29.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:29.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:29.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:29.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:29.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:29.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:29.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:29.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:29.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:29.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:29.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:29.340 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:29.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:29.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:29.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:29.341 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:29.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:29.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:29.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:29.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.437 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:29.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:29.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:29.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:29.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:29.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:29.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:29.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:29.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:29.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:29.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:29.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:29.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:29.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:29.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:29.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:29.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:29.442 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:29.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:29.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:29.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:29.443 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:29.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:29.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:29.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:29.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.506 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-262-gf9c422aad0 (f9c422aad058900772e144b1e747cee5fcb5727d) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xdf) [0x7fe63ab2f52c] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5eb) [0x55e5713c235b] 2021-02-06T13:08:29.507 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:03:55.867741+0000", , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 2.4185e-05 , 2021-02-06T03:09:06.897 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.507 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", , 2021-02-06T03:09:06.904 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:04:09.901372+0000", , 2021-02-06T03:09:06.905 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 5.1443000000000002e-05 2021-02-06T13:08:29.507 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 222.92658196299999, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.0022839280000000002, , 2021-02-06T03:09:06.912 INFO:teuthology.orchestra.run.gibba026.stdout: "type_data": { 2021-02-06T13:08:29.508 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "started", , 2021-02-06T03:09:06.920 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:05:56.633976+0000", 2021-02-06T13:08:29.508 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.928 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "all_read", 2021-02-06T13:08:29.509 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(client.34376.0:1909 2.2 e260/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head v 260'161, mlcod=252'157)", , 2021-02-06T03:09:06.935 INFO:teuthology.orchestra.run.gibba026.stdout: "initiated_at": "2021-02-06T03:06:32.703666+0000", , 2021-02-06T03:09:06.936 INFO:teuthology.orchestra.run.gibba026.stdout: "age": 154.18863902199999, 2021-02-06T13:08:29.509 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 8.8842000000000001e-05 , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.943 INFO:teuthology.orchestra.run.gibba026.stdout: { 2021-02-06T13:08:29.509 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:12.843965+0000", , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 0.00075681100000000003 , 2021-02-06T03:09:06.950 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.510 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: }, , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.958 INFO:teuthology.orchestra.run.gibba026.stdout: "description": "osd_repop(osd.5.0:214 2.2 e300/111 2:4b369c3d:::gibba04127561-3 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:b1 v 300'218, mlcod=300'215)", 2021-02-06T13:08:29.510 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "reached_pg", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:07:25.687929+0000", , 2021-02-06T03:09:06.965 INFO:teuthology.orchestra.run.gibba026.stdout: "duration": 9.9160000000000006e-05 2021-02-06T13:08:29.511 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.972 INFO:teuthology.orchestra.run.gibba026.stdout: { , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "event": "header_read", , 2021-02-06T03:09:06.973 INFO:teuthology.orchestra.run.gibba026.stdout: "time": "2021-02-06T03:08:34.647942+0000", 2021-02-06T13:08:29.511 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: ] , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: } , 2021-02-06T03:09:06.980 INFO:teuthology.orchestra.run.gibba026.stdout: }, 2021-02-06T13:08:29.512 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:09.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:08 gibba018 bash[14730]: cluster 2021-02-06T03:09:08.635744+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.648466+0000 osd.1 (osd.1) 38 : cluster [DBG] 1.13 scrub ok , 2021-02-06T03:09:10.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:09 gibba041 bash[14706]: cluster 2021-02-06T03:09:08.787956+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-06T13:08:29.512 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:14.386 DEBUG:teuthology.orchestra.run.gibba026:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.7 dump_historic_ops , 2021-02-06T03:09:14.466 DEBUG:teuthology.orchestra.run.gibba018:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph osd unset noscrub , Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.b/config 2021-02-06T13:08:29.513 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:24.944 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:24 gibba018 bash[14730]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.175 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:24 gibba026 bash[14737]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch , 2021-02-06T03:09:25.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:24 gibba041 bash[14706]: audit 2021-02-06T03:09:24.822415+0000 mon.a (mon.0) 1374 : audit [DBG] from='client.? v1:172.21.2.118:0/3648457110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-06T13:08:29.513 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:31.968852+0000 mon.a (mon.0) 1390 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch , 2021-02-06T03:09:33.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:09:32 gibba041 bash[14706]: audit 2021-02-06T03:09:32.041597+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14138 v1:172.21.2.118:0/3584536462' entity='mgr.y' , 2021-02-06T03:09:33.425 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:32 gibba026 bash[14737]: cluster 2021-02-06T03:09:31.954084+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e429: 12 total, 12 up, 12 in 2021-02-06T13:08:29.514 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.470218+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:09:41.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:40 gibba018 bash[14730]: cluster 2021-02-06T03:09:39.707876+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e434: 12 total, 12 up, 12 in , 2021-02-06T03:09:41.462 DEBUG:teuthology.orchestra.run.gibba041:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:caa58fce7a3c6a16d86ec2bf075a204e1e2b836c shell --fsid 9d5c7704-6826-11eb-8fda-001a4aab830c -- ceph daemon osd.11 dump_blocked_ops 2021-02-06T13:08:29.514 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:09:53.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:09:52 gibba018 bash[14730]: cluster 2021-02-06T03:09:51.810949+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e440: 12 total, 12 up, 12 in , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.589699+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:09:53.218 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:09:52 gibba026 bash[14737]: cluster 2021-02-06T03:09:51.590151+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-06T13:08:29.514 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.126086+0000 osd.5 (osd.5) 38 : cluster [DBG] 1.10 scrub ok , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.358478+0000 mgr.y (mgr.14138) 564 : cluster [DBG] pgmap v869: 37 pgs: 37 active+clean; 207 MiB data, 789 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:04.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:03 gibba018 bash[14730]: cluster 2021-02-06T03:10:02.408258+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-06T13:08:29.515 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/9d5c7704-6826-11eb-8fda-001a4aab830c/mon.a/config , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: cluster 2021-02-06T03:10:14.363578+0000 mgr.y (mgr.14138) 570 : cluster [DBG] pgmap v882: 37 pgs: 37 active+clean; 207 MiB data, 790 MiB used, 1007 GiB / 1008 GiB avail , 2021-02-06T03:10:16.057 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:15 gibba041 bash[14706]: audit 2021-02-06T03:10:14.710045+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.2.118:0/505213017' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-06T13:08:29.515 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695009+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: cluster 2021-02-06T03:10:29.695682+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok , 2021-02-06T03:10:30.957 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:30 gibba018 bash[14730]: audit 2021-02-06T03:10:30.483611+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? v1:172.21.2.118:0/4262438228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-06T13:08:29.516 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.515986+0000 osd.7 (osd.7) 51 : cluster [DBG] 1.2 scrub ok , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:40.797199+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e462: 12 total, 12 up, 12 in , 2021-02-06T03:10:42.176 INFO:journalctl@ceph.mon.b.gibba026.stdout:Feb 06 03:10:41 gibba026 bash[14737]: cluster 2021-02-06T03:10:41.083120+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2021-02-06T13:08:29.516 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: audit 2021-02-06T03:10:54.698103+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.2.118:0/1350565248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished , 2021-02-06T03:10:55.207 INFO:journalctl@ceph.mon.a.gibba018.stdout:Feb 06 03:10:54 gibba018 bash[14730]: cluster 2021-02-06T03:10:54.698191+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e468: 12 total, 12 up, 12 in , 2021-02-06T03:10:55.307 INFO:journalctl@ceph.mon.c.gibba041.stdout:Feb 06 03:10:54 gibba041 bash[14706]: cluster 2021-02-06T03:10:53.446121+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.4 scrub ok 2021-02-06T13:08:29.516 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f185165d85b8ba8477a192ef9ad9d44e8ec91644/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-02-06T13:08:29.517 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Got signal Terminated *** , 2021-02-06T03:11:23.457 INFO:journalctl@ceph.osd.2.gibba018.stdout:Feb 06 03:11:23 gibba018 bash[27123]: debug 2021-02-06T03:11:23.358+0000 7fd41def5700 -1 osd.2 470 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-06T13:08:29.517 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6807 osd.1 since back 2021-02-06T03:11:18.625877+0000 front 2021-02-06T03:11:18.625981+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:45.308 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:44 gibba041 bash[25645]: debug 2021-02-06T03:11:44.922+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6811 osd.2 since back 2021-02-06T03:11:20.926502+0000 front 2021-02-06T03:11:20.926359+0000 (oldest deadline 2021-02-06T03:11:43.826107+0000) , 2021-02-06T03:11:46.057 INFO:journalctl@ceph.osd.11.gibba041.stdout:Feb 06 03:11:45 gibba041 bash[25645]: debug 2021-02-06T03:11:45.938+0000 7fd7fb94d700 -1 osd.11 470 heartbeat_check: no reply from 172.21.2.118:6803 osd.0 since back 2021-02-06T03:11:16.325278+0000 front 2021-02-06T03:11:16.324629+0000 (oldest deadline 2021-02-06T03:11:38.624545+0000) 2021-02-06T13:08:29.517 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:============================================================================== , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:-hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.109 0.547 0.206 , 2021-02-06T03:12:00.327 INFO:teuthology.orchestra.run.gibba026.stdout:*hv02.front.sepi 163.237.218.18 2 u 40 64 377 0.131 -1.445 0.298 2021-02-06T13:08:29.518 WARNING:scrape:Ignoring malparsed backtrace: 2021-02-06T03:12:17.281 INFO:teuthology.task.internal:Tidying up after the test... , 2021-02-06T03:12:17.281 DEBUG:teuthology.orchestra.run.gibba018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-02-06T03:12:17.285 DEBUG:teuthology.orchestra.run.gibba026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-06T13:08:29.591 INFO:scrape:Found 10 distinct failure reasons 2021-02-06T13:08:29.592 INFO:scrape:Failure: "2021-02-05T22:30:28.931492+0000 mon.a (mon.0) 51 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2021-02-06T13:08:29.593 INFO:scrape:1 jobs: ['5849048'] 2021-02-06T13:08:29.593 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] 2021-02-06T13:08:29.593 INFO:scrape: 2021-02-06T13:08:29.593 INFO:scrape:Failure: Test failure: test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest) 2021-02-06T13:08:29.593 INFO:scrape:4 jobs: ['5849133', '5848992', '5849072', '5849153'] 2021-02-06T13:08:29.593 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{clusters/{2-node-mgr}', 'tasks/dashboard}'] 2021-02-06T13:08:29.593 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/dashboard}'] 2021-02-06T13:08:29.593 INFO:scrape: 2021-02-06T13:08:29.594 INFO:scrape:Failure: Scrubbing terminated -- not all pgs were active and clean. 2021-02-06T13:08:29.594 INFO:scrape:1 jobs: ['5849242'] 2021-02-06T13:08:29.594 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/lost-unfound-delete', 'supported-random-distro$/{ubuntu_latest}}'] 2021-02-06T13:08:29.594 INFO:scrape: 2021-02-06T13:08:29.594 INFO:scrape:Failure: Command failed on gibba001 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 72c72a48-67ae-11eb-8fd1-001a4aab830c -e sha1=caa58fce7a3c6a16d86ec2bf075a204e1e2b836c -- bash -c 'ceph orch ps'" 2021-02-06T13:08:29.594 INFO:scrape:2 jobs: ['5848917', '5849096'] 2021-02-06T13:08:29.594 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-02-06T13:08:29.595 INFO:scrape:suites union: ['1-tasks', 'distro1$/{rhel_latest}', 'distro1$/{ubuntu_20.04}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-02-06T13:08:29.595 INFO:scrape: 2021-02-06T13:08:29.595 INFO:scrape:Dead: b'2021-02-06T02:25:34.884 DEBUG:teuthology.exit:Finished running handlers' 2021-02-06T13:08:29.595 INFO:scrape:3 jobs: ['5848946', '5849057', '5849030'] 2021-02-06T13:08:29.595 INFO:scrape:suites intersection: [] 2021-02-06T13:08:29.595 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'openstack', 'openstack}', 'rados', 'rados/perf/{ceph', 'rados/verify/{centos_latest', 'scheduler/dmclock_default_shards', 'scheduler/wpq_default_shards', 'settings/optimized', 'tasks/rados_api_tests', 'thrashosds-health}', 'ubuntu_latest', 'validater/lockdep}', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}'] 2021-02-06T13:08:29.595 INFO:scrape: 2021-02-06T13:08:29.595 INFO:scrape:Failure: SSH connection to gibba041 was lost: 'sudo yum install -y kernel' 2021-02-06T13:08:29.596 INFO:scrape:2 jobs: ['5849125', '5849124'] 2021-02-06T13:08:29.596 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'objectstore/bluestore-stupid', 'openstack}', 'rados'] 2021-02-06T13:08:29.596 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/verify/{centos_latest', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'tasks/mon_recovery', 'thrashers/pggrow', 'thrashosds-health', 'validater/valgrind}', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2021-02-06T13:08:29.596 INFO:scrape: 2021-02-06T13:08:29.596 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on gibba044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=391cd0cd46833cb6ad17680e9cf8f36528f99432 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2021-02-06T13:08:29.596 INFO:scrape:2 jobs: ['5849045', '5848970'] 2021-02-06T13:08:29.596 INFO:scrape:suites intersection: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'rados'] 2021-02-06T13:08:29.597 INFO:scrape:suites union: ['clusters/3-mons', 'clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests}', 'thrashers/sync', 'workloads/rados_api_tests}'] 2021-02-06T13:08:29.597 INFO:scrape: 2021-02-06T13:08:29.597 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2021-02-06T13:08:29.597 INFO:scrape:1 jobs: ['5848906'] 2021-02-06T13:08:29.597 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'validater/valgrind}'] 2021-02-06T13:08:29.597 INFO:scrape: 2021-02-06T13:08:29.597 INFO:scrape:Crash: Test failure: test_pool_removal (tasks.mgr.test_progress.TestProgress) 2021-02-06T13:08:29.597 INFO:scrape:ceph version 17.0.0-339-gcaa58fce (caa58fce7a3c6a16d86ec2bf075a204e1e2b836c) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f024c3ff980] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1a7) [0x55dbe7a3049f] 5: (KernelDevice::_aio_thread()+0xbb9) [0x55dbe850f3a9] 6: (KernelDevice::AioCompletionThread::entry()+0xd) [0x55dbe851166d] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f024c3f46db] 8: clone() 2021-02-06T13:08:29.597 INFO:scrape:1 jobs: ['5848929'] 2021-02-06T13:08:29.598 INFO:scrape:suites: ['debug/mgr', 'mon_election/connectivity', 'objectstore/bluestore-hybrid', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/progress}'] 2021-02-06T13:08:29.598 INFO:scrape: 2021-02-06T13:08:29.598 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2021-02-06T13:08:29.598 INFO:scrape:1 jobs: ['5849127'] 2021-02-06T13:08:29.598 INFO:scrape:suites: ['1-install/nautilus-v1only', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/few', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/careful', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2021-02-06T13:08:29.598 INFO:scrape: 2021-02-06T13:08:29.614 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/ failed: 14 dead: 4 running: 0 waiting: 0 queued: 0 passed: 342 Fail ================================================================= [5848906] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:29:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848906/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848906/ reached maximum tries (50) after waiting for 300 seconds [5848917] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{rhel_latest} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:55:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848917/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9caa89fb8b024a07a9bbbaf9dea3b17c Command failed on gibba001 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 72c72a48-67ae-11eb-8fd1-001a4aab830c -e sha1=caa58fce7a3c6a16d86ec2bf075a204e1e2b836c -- bash -c 'ceph orch ps'" [5848929] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:13:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848929/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22e0089d286d44fe8c1b40c3eb3d7970 Test failure: test_pool_removal (tasks.mgr.test_progress.TestProgress) [5848970] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:32:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848970/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848970/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b701aa0b5d544cb9caf80e20d2aaffe Command failed (workunit test rados/test.sh) on gibba003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=391cd0cd46833cb6ad17680e9cf8f36528f99432 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5848992] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} ----------------------------------------------------------------- time: 00:57:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848992/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2287df980d4b49289358895e85b8c554 Test failure: test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest) [5849045] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:40:25 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849045/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849045/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a953bef05cb4904860700df74ff6494 Command failed (workunit test rados/test.sh) on gibba044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=391cd0cd46833cb6ad17680e9cf8f36528f99432 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5849048] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:08:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849048/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849048/ "2021-02-05T22:30:28.931492+0000 mon.a (mon.0) 51 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [5849072] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 00:52:21 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849072/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849072/ sentry: https://sentry.ceph.com/organizations/ceph/?query=144343b294274620a0e3d2a8e8fc9709 Test failure: test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest) [5849096] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:55:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849096/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849096/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b65e6bfa68f4ddd90d22b0106fce268 Command failed on gibba012 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a2f0820-6816-11eb-8fd7-001a4aab830c -e sha1=caa58fce7a3c6a16d86ec2bf075a204e1e2b836c -- bash -c 'ceph orch ps'" [5849124] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849124/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ff20cbe5b77468fa2c6968e217cddba Command failed on gibba026 with status 1: 'sudo yum install -y kernel' [5849127] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:21:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849127/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849127/ Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' [5849133] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 00:52:11 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849133/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849133/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75f750337acd4333bd0500833ec68509 Test failure: test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest) [5849153] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} ----------------------------------------------------------------- time: 00:51:54 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849153/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13546233fd84462d8ea989dcc5d73f08 Test failure: test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest) [5849242] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:02:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849242/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849242/ Scrubbing terminated -- not all pgs were active and clean. Dead ================================================================= [5848946] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848946/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848946/ [5849030] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849030/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849030/ [5849057] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849057/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849057/ [5849125] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849125/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849125/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eeae671cbcd54a45b8acf27b4ef133ba SSH connection to gibba041 was lost: 'sudo yum install -y kernel' Pass ================================================================= [5848899] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:28:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848899/ [5848900] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848900/ [5848901] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:57:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848901/ [5848902] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:10:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848902/ [5848903] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848903/ [5848904] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:52 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848904/ [5848905] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:34 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848905/ [5848907] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848907/ [5848908] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:30:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848908/ [5848909] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:16:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848909/ [5848910] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:05:36 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848910/ [5848911] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 01:07:10 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848911/ [5848912] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 00:54:03 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848912/ [5848913] rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}} time: 00:30:19 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848913/ [5848914] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:32:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848914/ [5848915] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:05 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848915/ [5848916] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush} time: 00:27:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848916/ [5848918] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:21:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848918/ [5848919] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:40:19 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848919/ [5848920] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:12:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848920/ [5848921] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:10:03 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848921/ [5848922] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:07:37 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848922/ [5848923] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:13:03 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848923/ [5848924] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:11:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848924/ [5848925] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:20:54 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848925/ [5848926] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:10:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848926/ [5848927] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848927/ [5848928] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} time: 00:15:08 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848928/ [5848930] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:13:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848930/ [5848931] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-radosbench} time: 03:17:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848931/ [5848932] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:12:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848932/ [5848933] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 00:54:16 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848933/ [5848934] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:43:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848934/ [5848935] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848935/ [5848936] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/many workloads/snaps-few-objects} time: 00:36:25 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848936/ [5848937] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:19:45 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848937/ [5848938] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:17:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848938/ [5848939] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:50:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848939/ [5848940] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848940/ [5848941] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:11:08 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848941/ [5848942] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:29:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848942/ [5848943] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848943/ [5848944] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:32 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848944/ [5848945] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:37 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848945/ [5848947] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848947/ [5848948] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:34:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848948/ [5848949] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:11:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848949/ [5848950] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:34:20 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848950/ [5848951] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:21:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848951/ [5848952] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:25:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848952/ [5848953] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:27:15 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848953/ [5848954] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/dashboard} time: 00:56:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848954/ [5848955] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:11:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848955/ [5848956] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:26:15 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848956/ [5848957] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:18:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848957/ [5848958] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:31:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848958/ [5848959] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:19:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848959/ [5848960] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:26:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848960/ [5848961] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848961/ [5848962] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:11:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848962/ [5848963] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr} time: 00:24:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848963/ [5848964] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848964/ [5848965] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:45:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848965/ [5848966] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848966/ [5848967] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:05:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848967/ [5848968] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848968/ [5848969] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:19:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848969/ [5848971] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/one workloads/pool-create-delete} time: 00:16:49 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848971/ [5848972] rados/singleton/{all/mon-config mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:22:01 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848972/ [5848973] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848973/ [5848974] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 00:57:09 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848974/ [5848975] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:02 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848975/ [5848976] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/connectivity} time: 00:18:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848976/ [5848977] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848977/ [5848978] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:17:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848978/ [5848979] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:29 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848979/ [5848980] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/valgrind} time: 01:48:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848980/ [5848981] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848981/ [5848982] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:41:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848982/ [5848983] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:09 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848983/ [5848984] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848984/ [5848985] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:24:10 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848985/ [5848986] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848986/ [5848987] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} time: 00:11:49 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848987/ [5848988] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:44:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848988/ [5848989] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/misc} time: 00:39:36 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848989/ [5848990] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:25:17 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848990/ [5848991] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848991/ [5848993] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} time: 00:07:16 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848993/ [5848994] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:25:16 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848994/ [5848995] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:19:15 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848995/ [5848996] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:20:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848996/ [5848997] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:18:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848997/ [5848998] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:27:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848998/ [5848999] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:14:41 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5848999/ [5849000] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:12:15 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849000/ [5849001] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849001/ [5849002] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} time: 00:22:28 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849002/ [5849003] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:52:38 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849003/ [5849004] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:29:32 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849004/ [5849005] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:17 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849005/ [5849006] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:09:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849006/ [5849007] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_5925} time: 00:08:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849007/ [5849008] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:12 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849008/ [5849009] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849009/ [5849010] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/rbd_cls} time: 00:21:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849010/ [5849011] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:11:03 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849011/ [5849012] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:08:07 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849012/ [5849013] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849013/ [5849014] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 00:56:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849014/ [5849015] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:08 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849015/ [5849016] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/lockdep} time: 00:11:39 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849016/ [5849017] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:45 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849017/ [5849018] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mon} time: 00:49:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849018/ [5849019] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:32:18 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849019/ [5849020] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/classic} time: 00:15:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849020/ [5849021] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:22 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849021/ [5849022] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:20:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849022/ [5849023] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:45:17 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849023/ [5849024] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:23:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849024/ [5849025] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:22:07 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849025/ [5849026] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:39:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849026/ [5849027] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:15:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849027/ [5849028] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/rados_python} time: 00:49:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849028/ [5849029] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:42:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849029/ [5849031] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/crash} time: 00:08:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849031/ [5849032] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:11:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849032/ [5849033] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 00:53:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849033/ [5849034] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:04:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849034/ [5849035] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:17:54 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849035/ [5849036] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:13:20 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849036/ [5849037] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:09:37 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849037/ [5849038] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:31:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849038/ [5849039] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:11:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849039/ [5849040] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849040/ [5849041] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:30:29 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849041/ [5849042] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:07:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849042/ [5849043] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:26:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849043/ [5849044] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} time: 03:19:07 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849044/ [5849046] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:45:01 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849046/ [5849047] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} time: 00:17:52 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849047/ [5849049] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind} time: 00:57:36 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849049/ [5849050] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:22:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849050/ [5849051] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:18:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849051/ [5849052] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:29 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849052/ [5849053] rados/singleton/{all/radostool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:25:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849053/ [5849054] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849054/ [5849055] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 00:52:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849055/ [5849056] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:23:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849056/ [5849058] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/connectivity} time: 00:15:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849058/ [5849059] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:27:28 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849059/ [5849060] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:18:05 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849060/ [5849061] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:18:10 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849061/ [5849062] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:21:21 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849062/ [5849063] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:19:11 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849063/ [5849064] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:28:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849064/ [5849065] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/failover} time: 00:11:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849065/ [5849066] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 01:48:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849066/ [5849067] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849067/ [5849068] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:21:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849068/ [5849069] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:36:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849069/ [5849070] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:18:32 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849070/ [5849071] rados/cephadm/smoke-roleless/{distro/ubuntu_20.04 start} time: 00:10:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849071/ [5849073] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{rhel_8}} time: 02:09:32 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849073/ [5849074] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:30 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849074/ [5849075] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:07:10 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849075/ [5849076] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} time: 00:21:49 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849076/ [5849077] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:33:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849077/ [5849078] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:09:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849078/ [5849079] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:56:18 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849079/ [5849080] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:16:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849080/ [5849081] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:19:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849081/ [5849082] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:20:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849082/ [5849083] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:23:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849083/ [5849084] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} time: 00:52:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849084/ [5849085] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:17 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849085/ [5849086] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/lockdep} time: 00:28:18 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849086/ [5849087] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:38 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849087/ [5849088] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:11:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849088/ [5849089] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:07:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849089/ [5849090] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:07:23 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849090/ [5849091] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:53:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849091/ [5849092] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:19:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849092/ [5849093] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 00:54:39 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849093/ [5849094] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:14:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849094/ [5849095] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/crush} time: 00:22:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849095/ [5849097] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache} time: 00:09:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849097/ [5849098] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:50:32 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849098/ [5849099] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:11:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849099/ [5849100] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}} time: 00:10:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849100/ [5849101] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:25:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849101/ [5849102] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/insights} time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849102/ [5849103] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:27:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849103/ [5849104] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:25 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849104/ [5849105] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:21:52 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849105/ [5849106] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:34 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849106/ [5849107] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:58 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849107/ [5849108] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:10:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849108/ [5849109] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:07:07 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849109/ [5849110] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849110/ [5849111] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 01:00:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849111/ [5849112] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/classic} time: 00:15:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849112/ [5849113] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 00:53:08 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849113/ [5849114] rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_8}} time: 01:04:20 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849114/ [5849115] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_workunits} time: 01:24:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849115/ [5849116] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849116/ [5849117] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 01:29:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849117/ [5849118] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 01:24:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849118/ [5849119] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/erasure-code} time: 01:34:38 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849119/ [5849120] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 01:28:23 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849120/ [5849121] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 01:05:39 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849121/ [5849122] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:57:18 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849122/ [5849123] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:02 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849123/ [5849126] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:06:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849126/ [5849128] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:37:22 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849128/ [5849129] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:02:25 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849129/ [5849130] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849130/ [5849131] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:37:41 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849131/ [5849132] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:35:56 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849132/ [5849134] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:20:41 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849134/ [5849135] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:22:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849135/ [5849136] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:33:01 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849136/ [5849137] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench} time: 02:01:58 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849137/ [5849138] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849138/ [5849139] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:18:09 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849139/ [5849140] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:34:18 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849140/ [5849141] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mgr} time: 00:25:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849141/ [5849142] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:07:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849142/ [5849143] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:11:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849143/ [5849144] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:10:18 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849144/ [5849145] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:24:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849145/ [5849146] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:07:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849146/ [5849147] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:10:41 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849147/ [5849148] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} time: 00:32:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849148/ [5849149] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:32:12 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849149/ [5849150] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:21:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849150/ [5849151] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:10:05 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849151/ [5849152] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} time: 00:14:52 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849152/ [5849154] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{rhel_8}} time: 00:12:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849154/ [5849155] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:16:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849155/ [5849156] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:24:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849156/ [5849157] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep} time: 00:37:23 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849157/ [5849158] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:06 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849158/ [5849159] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:20:01 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849159/ [5849160] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:23:02 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849160/ [5849161] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:11 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849161/ [5849162] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:30:05 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849162/ [5849163] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:08:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849163/ [5849164] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:13:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849164/ [5849165] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849165/ [5849166] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:07:12 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849166/ [5849167] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:35:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849167/ [5849168] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 01:10:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849168/ [5849169] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/progress} time: 00:19:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849169/ [5849170] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:02:37 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849170/ [5849171] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:06:29 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849171/ [5849172] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:16:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849172/ [5849173] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard} time: 00:52:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849173/ [5849174] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:10 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849174/ [5849175] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:10:16 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849175/ [5849176] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:19:08 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849176/ [5849177] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:45:38 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849177/ [5849178] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 01:02:22 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849178/ [5849179] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849179/ [5849180] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:40:22 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849180/ [5849181] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:04:58 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849181/ [5849182] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:22:05 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849182/ [5849183] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 01:12:07 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849183/ [5849184] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:15:11 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849184/ [5849185] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:10:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849185/ [5849186] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/pool-create-delete} time: 00:13:57 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849186/ [5849187] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:15 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849187/ [5849188] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:15:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849188/ [5849189] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:36 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849189/ [5849190] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:08:30 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849190/ [5849191] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:39 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849191/ [5849192] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} time: 00:56:20 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849192/ [5849193] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:11:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849193/ [5849194] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 00:53:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849194/ [5849195] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:36:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849195/ [5849196] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849196/ [5849197] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:19:50 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849197/ [5849198] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:56:01 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849198/ [5849199] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} time: 00:15:53 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849199/ [5849200] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:24:45 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849200/ [5849201] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:35:15 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849201/ [5849202] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:07:14 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849202/ [5849203] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/prometheus} time: 00:10:09 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849203/ [5849204] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:20:20 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849204/ [5849205] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:28 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849205/ [5849206] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:17:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849206/ [5849207] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:22:27 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849207/ [5849208] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:29:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849208/ [5849209] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:11 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849209/ [5849210] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:08:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849210/ [5849211] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:09:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849211/ [5849212] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:08:48 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849212/ [5849213] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849213/ [5849214] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 00:52:03 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849214/ [5849215] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:10:46 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849215/ [5849216] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849216/ [5849217] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 03:08:54 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849217/ [5849218] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:22:45 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849218/ [5849219] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:27:50 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849219/ [5849220] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:53:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849220/ [5849221] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 01:07:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849221/ [5849222] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:07:41 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849222/ [5849223] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/osd} time: 03:45:51 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849223/ [5849224] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 01:12:58 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849224/ [5849225] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:40 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849225/ [5849226] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 01:00:42 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849226/ [5849227] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:17:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849227/ [5849228] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849228/ [5849229] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:52 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849229/ [5849230] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:15:35 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849230/ [5849231] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:37:08 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849231/ [5849232] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:11:36 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849232/ [5849233] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/dashboard} time: 00:52:30 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849233/ [5849234] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_8}} time: 04:42:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849234/ [5849235] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:08:55 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849235/ [5849236] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/connectivity} time: 00:15:59 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849236/ [5849237] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:25:11 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849237/ [5849238] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:05 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849238/ [5849239] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:13:19 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849239/ [5849240] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:02 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849240/ [5849241] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:19:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849241/ [5849243] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:13 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849243/ [5849244] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:30:44 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849244/ [5849245] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/scrub} time: 01:50:29 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849245/ [5849246] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:26 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849246/ [5849247] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 01:09:49 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849247/ [5849248] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:25:04 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849248/ [5849249] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:37:34 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849249/ [5849250] rados/cephadm/smoke-roleless/{distro/ubuntu_20.04 start} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849250/ [5849251] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:22:29 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849251/ [5849252] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:17:24 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849252/ [5849253] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 00:49:41 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849253/ [5849254] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:31:45 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849254/ [5849255] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:00 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849255/ [5849256] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:35:02 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849256/ [5849257] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:34 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849257/ [5849258] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:16:47 info: http://pulpito.ceph.com/teuthology-2021-02-02_07:01:01-rados-master-distro-basic-gibba/5849258/