Found 367 jobs
Ignoring malparsed backtrace:  ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d]

Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch

Ignoring malparsed backtrace: nodeep-scrub is unset
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays

Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok
, 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok
, 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1
, Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument
, 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops

Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch
, 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in
, 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok
, 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished
, 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in
, 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok
, 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in
, 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y'
, 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1
, 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2...
, 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2

Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)
, 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000)
, 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

Ignoring malparsed backtrace:     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'
, 

Ignoring malparsed backtrace:  ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d]

Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch

Ignoring malparsed backtrace: nodeep-scrub is unset
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays

Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok
, 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok
, 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1
, Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument
, 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops

Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch
, 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in
, 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok
, 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished
, 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in
, 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok
, 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in
, 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y'
, 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1
, 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2...
, 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2

Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)
, 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000)
, 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

Ignoring malparsed backtrace:     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'
, 

Ignoring malparsed backtrace:  ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d]

Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch

Ignoring malparsed backtrace: nodeep-scrub is unset
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays

Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok
, 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok
, 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1
, Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument
, 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops

Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch
, 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in
, 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok
, 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished
, 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in
, 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok
, 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in
, 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y'
, 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1
, 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2...
, 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2

Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)
, 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000)
, 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

Ignoring malparsed backtrace:     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'
, 

Ignoring malparsed backtrace:  ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fe84b975d92]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe84b975f6d]

Ignoring malparsed backtrace: 2020-12-27T10:09:25.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.629773+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.163:0/1528678938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.754327+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch
, 2020-12-27T10:09:25.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:25 smithi168 bash[13415]: audit 2020-12-27T10:09:24.764280+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch

Ignoring malparsed backtrace: nodeep-scrub is unset
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.322583+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.1f scrub ok
, 2020-12-27T10:09:38.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:37 smithi153 bash[13362]: cluster 2020-12-27T10:09:36.807507+0000 mgr.y (mgr.14138) 627 : cluster [DBG] pgmap v1104: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:09:48.698 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:46.765898+0000 osd.4 (osd.4) 44 : cluster [DBG] 3.3 scrub ok
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.934405+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts
, 2020-12-27T10:09:48.699 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:48 smithi168 bash[13415]: cluster 2020-12-27T10:09:47.949113+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub found 4 strays

Ignoring malparsed backtrace: 2020-12-27T10:09:57.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:09:57 smithi153 bash[13362]: cluster 2020-12-27T10:09:57.014567+0000 osd.2 (osd.2) 44 : cluster [DBG] 1.25 scrub ok
, 2020-12-27T10:09:57.948 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.497134+0000 osd.7 (osd.7) 67 : cluster [DBG] 2.0 scrub ok
, 2020-12-27T10:09:57.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:09:57 smithi168 bash[13415]: cluster 2020-12-27T10:09:56.814765+0000 mgr.y (mgr.14138) 637 : cluster [DBG] pgmap v1124: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:10:08.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph osd primary-affinity 3 1
, Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:10:09.199 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:10:08 smithi168 bash[13415]: cluster 2020-12-27T10:10:07.871610+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:10:20.154 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument
, 2020-12-27T10:10:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
, 2020-12-27T10:10:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:3ef56c398ba628e9011753b67786b8e2a37447a6 shell --fsid 0adf27ae-4829-11eb-8f8d-001a4aab830c -- ceph daemon osd.11 dump_historic_ops

Ignoring malparsed backtrace: 2020-12-27T10:10:26.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:26 smithi163 bash[13296]: cluster 2020-12-27T10:10:25.975770+0000 osd.5 (osd.5) 66 : cluster [DBG] 1.10 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.224884+0000 osd.8 (osd.8) 41 : cluster [DBG] 1.29 scrub ok
, 2020-12-27T10:10:27.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:27 smithi153 bash[13362]: cluster 2020-12-27T10:10:26.259152+0000 osd.9 (osd.9) 43 : cluster [DBG] 1.a scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:10:38.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:10:38 smithi163 bash[13296]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch
, 2020-12-27T10:10:38.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: cluster 2020-12-27T10:10:37.390934+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in
, 2020-12-27T10:10:38.896 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:38 smithi153 bash[13362]: audit 2020-12-27T10:10:38.030940+0000 mon.a (mon.0) 1738 : audit [INF] from='client.? 172.21.15.163:0/3442788381' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:49.559552+0000 osd.4 (osd.4) 49 : cluster [DBG] 1.15 scrub ok
, 2020-12-27T10:10:51.647 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: audit 2020-12-27T10:10:50.345626+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.163:0/3265269229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished
, 2020-12-27T10:10:51.648 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:10:51 smithi153 bash[13362]: cluster 2020-12-27T10:10:50.345716+0000 mon.a (mon.0) 1751 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: Inferring config /var/lib/ceph/0adf27ae-4829-11eb-8f8d-001a4aab830c/mon.a/config
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: audit 2020-12-27T10:11:02.451240+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? 172.21.15.163:0/1429267878' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished
, 2020-12-27T10:11:03.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:03 smithi163 bash[13296]: cluster 2020-12-27T10:11:02.451361+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:11:15.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.492702+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in
, 2020-12-27T10:11:15.974 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:15 smithi163 bash[13296]: cluster 2020-12-27T10:11:14.844709+0000 mgr.y (mgr.14138) 679 : cluster [DBG] pgmap v1198: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:16.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:16 smithi153 bash[13362]: cluster 2020-12-27T10:11:14.880607+0000 osd.9 (osd.9) 44 : cluster [DBG] 1.26 scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:29.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:29 smithi163 bash[13296]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:27.844220+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.24 scrub ok
, 2020-12-27T10:11:29.646 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:29 smithi153 bash[13362]: cluster 2020-12-27T10:11:28.849774+0000 mgr.y (mgr.14138) 686 : cluster [DBG] pgmap v1208: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.626202+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.854727+0000 mgr.y (mgr.14138) 692 : cluster [DBG] pgmap v1217: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:11:41.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:41 smithi153 bash[13362]: cluster 2020-12-27T10:11:40.891199+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.19 deep-scrub ok

Ignoring malparsed backtrace: 2020-12-27T10:11:56.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.134563+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.2b scrub ok
, 2020-12-27T10:11:56.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:11:56 smithi163 bash[13296]: cluster 2020-12-27T10:11:55.384065+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in
, 2020-12-27T10:11:56.645 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:11:56 smithi153 bash[13362]: cluster 2020-12-27T10:11:54.744546+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts

Ignoring malparsed backtrace: 2020-12-27T10:12:12.895 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:12 smithi153 bash[13362]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.949 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:12 smithi168 bash[13415]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch
, 2020-12-27T10:12:12.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:12 smithi163 bash[13296]: audit 2020-12-27T10:12:12.496802+0000 mon.a (mon.0) 1793 : audit [DBG] from='client.? 172.21.15.163:0/3878852558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: cluster 2020-12-27T10:12:24.874687+0000 mgr.y (mgr.14138) 717 : cluster [DBG] pgmap v1248: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:25.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Dec 27 10:12:25 smithi153 bash[13362]: audit 2020-12-27T10:12:25.143408+0000 mon.a (mon.0) 1802 : audit [INF] from='mgr.14138 172.21.15.163:0/3144255808' entity='mgr.y'
, 2020-12-27T10:12:25.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:25 smithi168 bash[13415]: cluster 2020-12-27T10:12:24.638458+0000 mon.a (mon.0) 1799 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in

Ignoring malparsed backtrace: 2020-12-27T10:12:39.950 INFO:journalctl@ceph.mon.b.smithi168.stdout:Dec 27 10:12:39 smithi168 bash[13415]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.626766+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in
, 2020-12-27T10:12:39.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:39 smithi163 bash[13296]: cluster 2020-12-27T10:12:38.880753+0000 mgr.y (mgr.14138) 724 : cluster [DBG] pgmap v1257: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail

Ignoring malparsed backtrace: 2020-12-27T10:12:53.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.538822+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub ok
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: cluster 2020-12-27T10:12:52.886857+0000 mgr.y (mgr.14138) 731 : cluster [DBG] pgmap v1266: 59 pgs: 59 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail
, 2020-12-27T10:12:53.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Dec 27 10:12:53 smithi163 bash[13296]: audit 2020-12-27T10:12:52.965016+0000 mon.a (mon.0) 1814 : audit [DBG] from='client.? 172.21.15.163:0/460276682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch

Ignoring malparsed backtrace: 2020-12-27T10:13:04.864 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: 2020-12-27T10:13:25.903 INFO:tasks.cephadm.osd.1:Stopped osd.1
, 2020-12-27T10:13:25.904 INFO:tasks.cephadm.osd.2:Stopping osd.2...
, 2020-12-27T10:13:25.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-0adf27ae-4829-11eb-8f8d-001a4aab830c@osd.2

Ignoring malparsed backtrace: 2020-12-27T10:13:47.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:46 smithi153 bash[20221]: debug 2020-12-27T10:13:46.856+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2020-12-27T10:13:23.318960+0000 front 2020-12-27T10:13:23.317661+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)
, 2020-12-27T10:13:48.146 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2020-12-27T10:13:17.513944+0000 front 2020-12-27T10:13:17.515030+0000 (oldest deadline 2020-12-27T10:13:42.813596+0000)
, 2020-12-27T10:13:48.147 INFO:journalctl@ceph.osd.10.smithi153.stdout:Dec 27 10:13:47 smithi153 bash[20221]: debug 2020-12-27T10:13:47.808+0000 7f4a49e72700 -1 osd.10 665 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2020-12-27T10:13:22.815630+0000 front 2020-12-27T10:13:22.817101+0000 (oldest deadline 2020-12-27T10:13:46.817774+0000)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

Ignoring malparsed backtrace:     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'
, 

Found 7 distinct failure reasons
Failure: reached maximum tries (180) after waiting for 180 seconds
4 jobs: ['5739473', '5739613', '5739657', '5739380']
suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start']
suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']

Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2 jobs: ['5739575', '5739358']
suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']

Failure: Command crashed: 'CEPH_CLIENT_ID=2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'
1 jobs: ['5739432']
suites: ['1-install/nautilus-v2only', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}']

Dead: b'2020-12-27T21:59:32.409 DEBUG:teuthology.exit:Finished running handlers'
5 jobs: ['5739478', '5739620', '5739395', '5739434', '5739612']
suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health']
suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-radosbench}', 'workloads/ec-snaps-few-objects-overwrites}']

Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi095 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'
1 jobs: ['5739603']
suites: ['mon_election/classic', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}']

Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ef56c398ba628e9011753b67786b8e2a37447a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2 jobs: ['5739372', '5739497']
suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']
suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']

Failure: psutil.NoSuchProcess process no longer exists (pid=13711)
1 jobs: ['5739411']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/pool-snaps-few-objects}']