Found 367 jobs Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Ignoring malparsed backtrace: ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f28f605f826] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6] Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.102176+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", , 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:24.500852+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "op_commit", , 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.650960+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", , 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:30.575183+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: ] , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: } , 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "started", Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { , 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 6.2530000000000001e-06 , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout: "type_data": { , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout: "client_info": { Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:49:47.301027+0000", , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.81516294099999997 , 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "queued_for_pg", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.400750+0000", , 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.2364999999999998e-05 Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "all_read", , 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:47.622931+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout: "client": "client.4431", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "client_addr": "172.21.15.93:0/2543268328", , 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout: "tid": 4001 Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:48:41.525747+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 7.8399999999999995e-06 , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)", , 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:14.150340+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:25.319281+0000", , 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.00012395200000000001 , 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout: }, Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "dispatched", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.898369+0000", , 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 4.9640000000000002e-06 Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "throttled", , 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.899108+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "reached_pg", , 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout: "time": "2020-12-31T07:50:41.906406+0000", Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: { , 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout: "event": "header_read", Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout: "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "initiated_at": "2020-12-31T07:50:52.246307+0000", , 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout: "age": 6.3414430709999996, Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout: "duration": 0.97759732200000005 , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: }, , 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout: { Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument , 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head# , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{} , 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head# Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 , 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure , set osd.1 primary-affinity to 1 (8655362) Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{ , 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task , running.get() Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs... Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump , 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core Found 10 distinct failure reasons Failure: reached maximum tries (180) after waiting for 180 seconds 4 jobs: ['5749005', '5749038', '5748945', '5748907'] suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] Failure: Command failed on smithi037 with status 1: 'sudo docker kill -s 1 ceph-1e2c34cc-4b45-11eb-8f8d-001a4aab830c-osd.10' 1 jobs: ['5748845'] suites: ['1-install/mimic', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['5748816', '5748960'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] Dead: b'2020-12-31T21:32:11.994 DEBUG:teuthology.exit:Finished running handlers' 4 jobs: ['5748926', '5748998', '5748856', '5749129'] suites intersection: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'openstack}', 'rados', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-small}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 1 jobs: ['5749032'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] Failure: Command failed (workunit test cephtool/test.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 1 jobs: ['5748890'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rhel_8}}'] Timeout 120 running ceph --cluster ceph pg dump --format=json 1 jobs: ['5748847'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/max-pg-per-osd.from-mon', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 1 jobs: ['5748855'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] Failure: Command failed on smithi092 with status 1: 'sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump' 2 jobs: ['5748961', '5749029'] suites intersection: [] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/singleton/{all/resolve_stuck_peering', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'tasks/dashboard}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 50 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 1 jobs: ['5748785'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect_promote_tests}']