2020-12-27T08:28:48.353 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-27T08:28:48.355 INFO:teuthology.results:subset: '0/100000' 2020-12-27T08:28:48.355 INFO:teuthology.results:seed: '9365' 2020-12-27T11:31:36.187 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-27T11:31:36.223 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-27T23:05:04.040 INFO:teuthology.results:Tests finished! gathering results... 2020-12-27T23:05:41.006 INFO:scrape:Found 367 jobs 2020-12-27T23:05:43.497 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d] 2020-12-27T23:05:43.498 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2020-12-27T23:05:43.498 WARNING:scrape:Ignoring malparsed backtrace: nodeep-scrub is unset , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.499 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays 2020-12-27T23:05:43.499 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok , 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok , 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.500 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1 , Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.501 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument , 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops 2020-12-27T23:05:43.501 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok 2020-12-27T23:05:43.502 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch , 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in , 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2020-12-27T23:05:43.502 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok , 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished , 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2020-12-27T23:05:43.503 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2020-12-27T23:05:43.504 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in , 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok 2020-12-27T23:05:43.504 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.505 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok 2020-12-27T23:05:43.506 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok , 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in , 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.506 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2020-12-27T23:05:43.507 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' , 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2020-12-27T23:05:43.507 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.509 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2020-12-27T23:05:43.510 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested 2020-12-27T23:05:43.510 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1 , 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2... , 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2 2020-12-27T23:05:43.511 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) , 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000) , 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) 2020-12-27T23:05:43.512 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 2020-12-27T23:05:43.512 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' , 2020-12-27T23:05:43.608 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d] 2020-12-27T23:05:43.609 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2020-12-27T23:05:43.609 WARNING:scrape:Ignoring malparsed backtrace: nodeep-scrub is unset , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays 2020-12-27T23:05:43.611 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok , 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok , 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.611 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1 , Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument , 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops 2020-12-27T23:05:43.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok 2020-12-27T23:05:43.613 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch , 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in , 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2020-12-27T23:05:43.614 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok , 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished , 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2020-12-27T23:05:43.614 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2020-12-27T23:05:43.615 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in , 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok 2020-12-27T23:05:43.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok 2020-12-27T23:05:43.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok , 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in , 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2020-12-27T23:05:43.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' , 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2020-12-27T23:05:43.619 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2020-12-27T23:05:43.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested 2020-12-27T23:05:43.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1 , 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2... , 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2 2020-12-27T23:05:43.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) , 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000) , 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) 2020-12-27T23:05:43.623 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 2020-12-27T23:05:43.624 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' , 2020-12-27T23:05:43.717 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d] 2020-12-27T23:05:43.718 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2020-12-27T23:05:43.719 WARNING:scrape:Ignoring malparsed backtrace: nodeep-scrub is unset , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.719 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays 2020-12-27T23:05:43.720 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok , 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok , 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.721 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1 , Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.721 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument , 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops 2020-12-27T23:05:43.722 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok 2020-12-27T23:05:43.723 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch , 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in , 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2020-12-27T23:05:43.723 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok , 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished , 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2020-12-27T23:05:43.724 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2020-12-27T23:05:43.725 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in , 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok 2020-12-27T23:05:43.725 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.726 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok 2020-12-27T23:05:43.727 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok , 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in , 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.727 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2020-12-27T23:05:43.728 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' , 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2020-12-27T23:05:43.728 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.730 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2020-12-27T23:05:43.731 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested 2020-12-27T23:05:43.731 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1 , 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2... , 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2 2020-12-27T23:05:43.732 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) , 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000) , 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) 2020-12-27T23:05:43.733 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 2020-12-27T23:05:43.733 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' , 2020-12-27T23:05:43.828 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d] 2020-12-27T23:05:43.829 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch , 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2020-12-27T23:05:43.830 WARNING:scrape:Ignoring malparsed backtrace: nodeep-scrub is unset , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok , 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.831 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts , 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays 2020-12-27T23:05:43.831 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok , 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok , 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.832 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1 , Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.833 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument , 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops 2020-12-27T23:05:43.833 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok , 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok 2020-12-27T23:05:43.834 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch , 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in , 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2020-12-27T23:05:43.835 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok , 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished , 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2020-12-27T23:05:43.835 WARNING:scrape:Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished , 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2020-12-27T23:05:43.836 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in , 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok 2020-12-27T23:05:43.837 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok , 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.837 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok 2020-12-27T23:05:43.838 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok , 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in , 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2020-12-27T23:05:43.838 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch , 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2020-12-27T23:05:43.839 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' , 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2020-12-27T23:05:43.840 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in , 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2020-12-27T23:05:43.841 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail , 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2020-12-27T23:05:43.842 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested 2020-12-27T23:05:43.843 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1 , 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2... , 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2 2020-12-27T23:05:43.843 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) , 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000) , 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000) 2020-12-27T23:05:43.844 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 2020-12-27T23:05:43.845 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' , 2020-12-27T23:05:44.691 INFO:scrape:Found 7 distinct failure reasons 2020-12-27T23:05:44.692 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-12-27T23:05:44.693 INFO:scrape:4 jobs: ['5739473', '5739613', '5739657', '5739380'] 2020-12-27T23:05:44.693 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-12-27T23:05:44.694 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-12-27T23:05:44.694 INFO:scrape: 2020-12-27T23:05:44.695 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2020-12-27T23:05:44.695 INFO:scrape:2 jobs: ['5739575', '5739358'] 2020-12-27T23:05:44.695 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2020-12-27T23:05:44.696 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2020-12-27T23:05:44.696 INFO:scrape: 2020-12-27T23:05:44.697 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 2020-12-27T23:05:44.697 INFO:scrape:1 jobs: ['5739432'] 2020-12-27T23:05:44.698 INFO:scrape:suites: ['1-install/nautilus-v2only', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}'] 2020-12-27T23:05:44.698 INFO:scrape: 2020-12-27T23:05:44.699 INFO:scrape:Dead: b'2020-12-27T21:59:32.409 DEBUG:teuthology.exit:Finished running handlers' 2020-12-27T23:05:44.699 INFO:scrape:5 jobs: ['5739478', '5739620', '5739395', '5739434', '5739612'] 2020-12-27T23:05:44.699 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2020-12-27T23:05:44.700 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-radosbench}', 'workloads/ec-snaps-few-objects-overwrites}'] 2020-12-27T23:05:44.700 INFO:scrape: 2020-12-27T23:05:44.701 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi095 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-12-27T23:05:44.701 INFO:scrape:1 jobs: ['5739603'] 2020-12-27T23:05:44.702 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-12-27T23:05:44.702 INFO:scrape: 2020-12-27T23:05:44.702 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-12-27T23:05:44.703 INFO:scrape:2 jobs: ['5739372', '5739497'] 2020-12-27T23:05:44.703 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-27T23:05:44.704 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-27T23:05:44.704 INFO:scrape: 2020-12-27T23:05:44.705 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=13711) 2020-12-27T23:05:44.705 INFO:scrape:1 jobs: ['5739411'] 2020-12-27T23:05:44.705 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/pool-snaps-few-objects}'] 2020-12-27T23:05:44.706 INFO:scrape: 2020-12-27T23:05:44.718 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/ failed: 10 dead: 6 running: 0 waiting: 0 queued: 0 passed: 351 Fail ================================================================= [5739358] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:46:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739358/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7f70e74949d4060b5b056c32e158c0a Command failed (workunit test rados/test.sh) on smithi190 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5739372] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:15:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739372/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81471b1108c04a6fa596ef6e16bc553f Command failed (workunit test cephadm/test_cephadm.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5739380] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:44 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739380/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f261a8a53bdc420883ea315f23ed81f3 reached maximum tries (180) after waiting for 180 seconds [5739432] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:27:48 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739432/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739432/ Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' [5739473] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:17:08 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739473/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739473/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96707837a40042679382548a7474609d reached maximum tries (180) after waiting for 180 seconds [5739497] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:15:05 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739497/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739497/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c3c0d1fe6045588d9cad38603afb5f Command failed (workunit test cephadm/test_cephadm.sh) on smithi189 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5739575] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:47:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739575/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ae6884f871d4c2da9b066a617a27e16 Command failed (workunit test rados/test.sh) on smithi146 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5739603] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739603/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739603/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7098ad88ce7845778d3206ef4e244f15 Command failed (workunit test cephadm/test_adoption.sh) on smithi095 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5739613] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:18:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739613/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739613/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04149e33cfd64647b9a8d4838147ee79 reached maximum tries (180) after waiting for 180 seconds [5739657] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:51 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739657/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4414ef8addea4576b40e28ffbbac7d50 reached maximum tries (180) after waiting for 180 seconds Dead ================================================================= [5739395] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739395/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739395/ [5739411] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 11:48:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739411/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739411/ psutil.NoSuchProcess process no longer exists (pid=13711) [5739434] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739434/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739434/ [5739478] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739478/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739478/ [5739612] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739612/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739612/ [5739620] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739620/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739620/ Pass ================================================================= [5739311] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:33:02 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739311/ [5739312] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:31:59 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739312/ [5739313] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:55 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739313/ [5739314] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:08:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739314/ [5739315] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/pool-create-delete} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739315/ [5739316] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:09:09 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739316/ [5739317] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:16:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739317/ [5739318] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:18:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739318/ [5739319] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739319/ [5739320] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:08:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739320/ [5739321] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:43:50 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739321/ [5739322] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:16:01 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739322/ [5739323] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/crush} time: 00:19:15 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739323/ [5739324] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:34:38 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739324/ [5739325] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:25:55 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739325/ [5739326] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:32:43 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739326/ [5739327] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:11 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739327/ [5739328] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:28:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739328/ [5739329] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:47 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739329/ [5739330] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 05:17:17 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739330/ [5739331] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739331/ [5739332] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/lockdep} time: 00:12:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739332/ [5739333] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:13:09 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739333/ [5739334] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:33 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739334/ [5739335] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:16:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739335/ [5739336] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:14:25 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739336/ [5739337] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:26 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739337/ [5739338] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big} time: 00:44:48 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739338/ [5739339] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:38:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739339/ [5739340] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:17:23 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739340/ [5739341] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:10:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739341/ [5739342] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:26 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739342/ [5739343] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:35 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739343/ [5739344] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:39:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739344/ [5739345] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} time: 00:14:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739345/ [5739346] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:21:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739346/ [5739347] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:41:26 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739347/ [5739348] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:16:13 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739348/ [5739349] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739349/ [5739350] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_cls_all} time: 00:24:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739350/ [5739351] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:08:04 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739351/ [5739352] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739352/ [5739353] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench} time: 01:24:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739353/ [5739354] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:13 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739354/ [5739355] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:23:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739355/ [5739356] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/failover} time: 00:13:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739356/ [5739357] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:59 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739357/ [5739359] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739359/ [5739360] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:27 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739360/ [5739361] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:26:48 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739361/ [5739362] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_5925} time: 00:07:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739362/ [5739363] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:28:04 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739363/ [5739364] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:10:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739364/ [5739365] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:31 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739365/ [5739366] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:10:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739366/ [5739367] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:14:55 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739367/ [5739368] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:18 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739368/ [5739369] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739369/ [5739370] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps} time: 00:18:09 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739370/ [5739371] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739371/ [5739373] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:37:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739373/ [5739374] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:23:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739374/ [5739375] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mgr} time: 00:18:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739375/ [5739376] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:11 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739376/ [5739377] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739377/ [5739378] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:19:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739378/ [5739379] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:25:40 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739379/ [5739381] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/rados_python} time: 00:23:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739381/ [5739382] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:15:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739382/ [5739383] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:53 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739383/ [5739384] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:42:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739384/ [5739385] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739385/ [5739386] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739386/ [5739387] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:29 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739387/ [5739388] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:24:08 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739388/ [5739389] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739389/ [5739390] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/insights} time: 00:13:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739390/ [5739391] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:45:50 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739391/ [5739392] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739392/ [5739393] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:20 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739393/ [5739394] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:28:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739394/ [5739396] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:10:27 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739396/ [5739397] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:36:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739397/ [5739398] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_api_tests} time: 00:37:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739398/ [5739399] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:35:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739399/ [5739400] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:15:13 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739400/ [5739401] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/misc} time: 00:35:38 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739401/ [5739402] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739402/ [5739403] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache} time: 00:13:15 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739403/ [5739404] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:25:18 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739404/ [5739405] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:46 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739405/ [5739406] rados/objectstore/{backends/filejournal supported-random-distro$/{rhel_8}} time: 00:09:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739406/ [5739407] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:13:38 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739407/ [5739408] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739408/ [5739409] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:28:41 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739409/ [5739410] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739410/ [5739412] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:08:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739412/ [5739413] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:24:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739413/ [5739414] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:05:55 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739414/ [5739415] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:43:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739415/ [5739416] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:40 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739416/ [5739417] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:29 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739417/ [5739418] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739418/ [5739419] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739419/ [5739420] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739420/ [5739421] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739421/ [5739422] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739422/ [5739423] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:36:17 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739423/ [5739424] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:15 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739424/ [5739425] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:28:59 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739425/ [5739426] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:50:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739426/ [5739427] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:17:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739427/ [5739428] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739428/ [5739429] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739429/ [5739430] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} time: 00:27:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739430/ [5739431] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739431/ [5739433] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:06:18 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739433/ [5739435] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:53:04 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739435/ [5739436] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:09:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739436/ [5739437] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:39:09 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739437/ [5739438] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:54:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739438/ [5739439] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739439/ [5739440] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} time: 00:17:11 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739440/ [5739441] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739441/ [5739442] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739442/ [5739443] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:09:44 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739443/ [5739444] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:17:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739444/ [5739445] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739445/ [5739446] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:26:11 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739446/ [5739447] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:20:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739447/ [5739448] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739448/ [5739449] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:40:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739449/ [5739450] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:14:53 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739450/ [5739451] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:22:31 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739451/ [5739452] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:17:25 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739452/ [5739453] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:19:46 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739453/ [5739454] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:18:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739454/ [5739455] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:29:51 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739455/ [5739456] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:20 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739456/ [5739457] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect} time: 00:15:40 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739457/ [5739458] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:09:25 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739458/ [5739459] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739459/ [5739460] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/progress} time: 00:23:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739460/ [5739461] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:43 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739461/ [5739462] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:21:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739462/ [5739463] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:08:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739463/ [5739464] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:18:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739464/ [5739465] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739465/ [5739466] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:32:08 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739466/ [5739467] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739467/ [5739468] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739468/ [5739469] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:24:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739469/ [5739470] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_workunits} time: 00:31:17 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739470/ [5739471] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:15:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739471/ [5739472] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:16:19 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739472/ [5739474] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:16:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739474/ [5739475] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:39:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739475/ [5739476] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:17:27 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739476/ [5739477] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:20:35 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739477/ [5739479] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/scrub} time: 01:45:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739479/ [5739480] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739480/ [5739481] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:12:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739481/ [5739482] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:11:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739482/ [5739483] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:25:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739483/ [5739484] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:40:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739484/ [5739485] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:19:48 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739485/ [5739486] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:33 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739486/ [5739487] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:29:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739487/ [5739488] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:15 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739488/ [5739489] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739489/ [5739490] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:01 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739490/ [5739491] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:16:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739491/ [5739492] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739492/ [5739493] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:18:13 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739493/ [5739494] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:45:59 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739494/ [5739495] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/prometheus} time: 00:11:35 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739495/ [5739496] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:12:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739496/ [5739498] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739498/ [5739499] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} time: 02:21:20 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739499/ [5739500] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:39 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739500/ [5739501] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739501/ [5739502] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:11:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739502/ [5739503] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:20:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739503/ [5739504] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739504/ [5739505] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:30:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739505/ [5739506] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:28:41 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739506/ [5739507] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:15:11 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739507/ [5739508] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:44:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739508/ [5739509] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739509/ [5739510] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739510/ [5739511] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/crush} time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739511/ [5739512] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} time: 05:02:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739512/ [5739513] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:29:07 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739513/ [5739514] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739514/ [5739515] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:32:05 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739515/ [5739516] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:06:20 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739516/ [5739517] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:21:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739517/ [5739518] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:17:44 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739518/ [5739519] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:12:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739519/ [5739520] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739520/ [5739521] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:12:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739521/ [5739522] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739522/ [5739523] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:07:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739523/ [5739524] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:12:17 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739524/ [5739525] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739525/ [5739526] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:35 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739526/ [5739527] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:24:17 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739527/ [5739528] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739528/ [5739529] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/classic} time: 00:17:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739529/ [5739530] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:31:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739530/ [5739531] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/workunits} time: 00:16:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739531/ [5739532] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:45:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739532/ [5739533] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:04 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739533/ [5739534] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:38:59 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739534/ [5739535] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:19:35 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739535/ [5739536] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:25:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739536/ [5739537] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:48 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739537/ [5739538] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:43 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739538/ [5739539] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:08 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739539/ [5739540] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739540/ [5739541] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:23 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739541/ [5739542] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:42:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739542/ [5739543] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:05:33 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739543/ [5739544] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:26:40 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739544/ [5739545] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/many workloads/pool-create-delete} time: 00:18:00 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739545/ [5739546] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:13:25 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739546/ [5739547] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:06:05 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739547/ [5739548] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739548/ [5739549] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:14:51 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739549/ [5739550] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739550/ [5739551] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:37 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739551/ [5739552] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:41 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739552/ [5739553] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739553/ [5739554] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:35:41 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739554/ [5739555] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:22:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739555/ [5739556] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mgr} time: 00:18:29 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739556/ [5739557] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:08 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739557/ [5739558] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:08:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739558/ [5739559] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739559/ [5739560] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:28:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739560/ [5739561] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/connectivity} time: 00:17:41 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739561/ [5739562] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:12:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739562/ [5739563] rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:30:50 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739563/ [5739564] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:08:47 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739564/ [5739565] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:10:07 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739565/ [5739566] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}} time: 00:13:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739566/ [5739567] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:20:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739567/ [5739568] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:28:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739568/ [5739569] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739569/ [5739570] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739570/ [5739571] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:10:44 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739571/ [5739572] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:08:05 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739572/ [5739573] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:16:27 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739573/ [5739574] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:28 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739574/ [5739576] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:04 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739576/ [5739577] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:15:50 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739577/ [5739578] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:18 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739578/ [5739579] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects} time: 00:18:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739579/ [5739580] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} time: 00:07:09 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739580/ [5739581] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/rbd_cls} time: 00:20:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739581/ [5739582] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:58 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739582/ [5739583] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} time: 00:17:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739583/ [5739584] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:24:17 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739584/ [5739585] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:32:27 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739585/ [5739586] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:30:44 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739586/ [5739587] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739587/ [5739588] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:17:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739588/ [5739589] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:18 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739589/ [5739590] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:38:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739590/ [5739591] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:17:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739591/ [5739592] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:35:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739592/ [5739593] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739593/ [5739594] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:23:46 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739594/ [5739595] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:21:08 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739595/ [5739596] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739596/ [5739597] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:08:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739597/ [5739598] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:16 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739598/ [5739599] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:45:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739599/ [5739600] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:16:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739600/ [5739601] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/failover} time: 00:12:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739601/ [5739602] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:51 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739602/ [5739604] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:15:05 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739604/ [5739605] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:18:43 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739605/ [5739606] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:19 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739606/ [5739607] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:34:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739607/ [5739608] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739608/ [5739609] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mon} time: 00:51:25 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739609/ [5739610] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739610/ [5739611] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:24:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739611/ [5739614] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:14:43 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739614/ [5739615] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:21:32 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739615/ [5739616] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:53 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739616/ [5739617] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_api_tests} time: 00:31:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739617/ [5739618] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:28:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739618/ [5739619] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:24:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739619/ [5739621] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:11:53 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739621/ [5739622] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739622/ [5739623] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:12:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739623/ [5739624] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:39 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739624/ [5739625] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:42 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739625/ [5739626] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:08:57 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739626/ [5739627] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 01:16:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739627/ [5739628] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739628/ [5739629] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:38 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739629/ [5739630] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:25:30 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739630/ [5739631] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:30:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739631/ [5739632] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:03 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739632/ [5739633] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:21:54 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739633/ [5739634] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-overwrites} time: 00:21:20 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739634/ [5739635] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/insights} time: 00:13:05 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739635/ [5739636] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:15:24 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739636/ [5739637] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:32:59 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739637/ [5739638] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:40:19 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739638/ [5739639] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:21:29 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739639/ [5739640] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:09:18 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739640/ [5739641] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:21:13 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739641/ [5739642] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:31 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739642/ [5739643] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:12 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739643/ [5739644] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:27:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739644/ [5739645] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:52 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739645/ [5739646] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} time: 02:16:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739646/ [5739647] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:22 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739647/ [5739648] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} time: 05:00:21 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739648/ [5739649] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:33:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739649/ [5739650] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:14:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739650/ [5739651] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:08:38 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739651/ [5739652] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:25:10 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739652/ [5739653] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:57:34 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739653/ [5739654] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:31:04 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739654/ [5739655] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/radosbench} time: 01:02:48 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739655/ [5739656] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:15:35 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739656/ [5739658] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:30:55 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739658/ [5739659] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:24:43 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739659/ [5739660] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739660/ [5739661] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:42 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739661/ [5739662] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/scrub} time: 01:45:49 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739662/ [5739663] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache} time: 00:12:13 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739663/ [5739664] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:22:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739664/ [5739665] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:28:06 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739665/ [5739666] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:28:36 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739666/ [5739667] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:09:45 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739667/ [5739668] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:40 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739668/ [5739669] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:19:56 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739669/ [5739670] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:25:50 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739670/ [5739671] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:15 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739671/ [5739672] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:31:40 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739672/ [5739673] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739673/ [5739674] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:16:14 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739674/ [5739675] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:05:41 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739675/ [5739676] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm_repos} time: 00:11:53 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739676/ [5739677] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:12:44 info: http://pulpito.ceph.com/teuthology-2020-12-27_07:01:02-rados-master-distro-basic-smithi/5739677/