2020-12-31T07:31:00.589 INFO:root:teuthology version: 1.0.0-8e49dda4
2020-12-31T07:31:00.591 INFO:teuthology.results:subset: '4/100000'
2020-12-31T07:31:00.591 INFO:teuthology.results:seed: '6677'
2020-12-31T11:46:29.899 INFO:root:teuthology version: 1.0.0-8e49dda4
2020-12-31T11:46:30.006 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-12-31T23:37:55.156 INFO:teuthology.results:Tests finished! gathering results...
2020-12-31T23:39:05.944 INFO:scrape:Found 367 jobs
2020-12-31T23:39:16.753 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:16.754 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:16.755 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:16.756 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:16.756 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:16.757 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:16.758 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:16.758 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:16.759 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:16.759 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:16.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:16.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:16.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:16.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:16.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:16.763 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:16.764 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:16.764 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:16.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:16.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:16.766 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:16.766 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:16.767 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:16.768 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:16.768 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:16.769 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:16.769 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:16.770 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:16.771 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:16.771 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:16.772 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:16.774 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:16.775 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:16.776 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:16.776 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:16.777 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:16.944 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:16.945 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:16.946 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:16.946 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:16.947 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:16.947 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:16.948 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:16.948 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:16.949 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:16.950 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:16.950 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:16.951 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:16.951 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:16.952 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:16.952 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:16.953 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:16.954 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:16.954 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:16.955 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:16.955 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:16.956 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:16.957 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:16.957 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:16.958 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:16.958 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:16.959 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:16.959 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:16.960 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:16.961 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:16.961 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:16.962 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:16.964 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:16.965 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:16.966 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:16.966 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:16.967 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:17.121 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:17.122 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:17.123 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:17.124 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:17.124 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:17.125 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:17.125 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:17.126 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:17.126 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:17.127 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:17.127 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:17.128 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.130 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:17.131 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.132 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:17.133 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:17.133 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:17.134 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:17.134 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.135 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:17.136 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.136 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:17.137 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.137 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:17.138 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:17.138 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:17.139 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.140 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:17.140 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.141 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:17.142 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:17.143 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:17.145 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:17.145 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:17.146 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:17.147 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:17.288 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:17.289 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:17.290 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:17.291 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:17.291 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:17.292 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:17.293 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:17.293 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:17.294 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:17.294 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:17.295 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:17.295 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.296 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:17.297 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.297 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:17.298 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:17.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:17.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:17.300 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.301 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:17.301 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:17.302 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.303 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:17.304 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:17.304 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:17.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.305 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:17.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.306 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:17.307 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:17.309 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:17.311 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:17.311 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:17.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:17.312 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:17.447 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:17.448 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:17.448 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:17.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:17.449 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:17.450 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:17.451 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:17.451 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:17.452 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:17.452 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:17.453 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:17.453 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.454 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:17.455 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.455 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:17.456 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:17.456 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:17.457 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:17.457 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.458 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:17.458 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.459 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:17.460 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.460 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:17.461 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:17.461 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:17.462 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.462 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:17.463 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.464 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:17.464 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:17.466 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:17.467 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:17.468 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:17.469 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:17.469 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:17.606 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:17.607 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:17.607 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:17.608 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:17.608 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:17.609 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:17.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:17.610 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:17.611 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:17.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:17.612 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:17.613 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.613 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:17.614 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.614 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:17.615 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:17.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:17.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:17.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:17.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.619 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:17.619 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.620 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:17.620 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:17.621 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:17.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:17.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.623 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:17.624 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:17.626 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:17.627 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:17.628 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:17.629 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:17.629 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:17.769 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:17.770 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:17.771 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:17.772 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:17.772 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:17.773 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:17.773 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:17.774 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:17.775 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:17.775 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:17.776 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:17.776 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.777 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:17.778 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.778 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:17.779 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:17.780 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:17.780 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:17.781 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.781 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:17.782 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.783 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:17.783 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.784 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:17.784 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:17.785 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:17.786 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.786 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:17.787 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.787 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:17.788 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:17.790 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:17.791 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:17.792 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:17.793 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:17.793 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:17.934 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:17.935 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:17.936 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:17.936 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:17.937 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:17.937 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:17.938 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:17.939 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:17.939 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:17.940 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:17.940 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:17.941 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.941 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:17.942 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.943 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:17.943 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:17.944 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:17.944 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:17.945 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.946 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:17.946 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.947 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:17.947 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:17.948 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:17.948 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:17.949 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:17.950 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:17.950 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:17.951 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:17.951 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:17.952 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:17.954 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:17.955 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:17.956 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:17.957 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:17.958 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:18.096 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 16.0.0-8610-g1bfac618 (1bfac6181a18be3417e8090e0e2894c39783abc1) pacific (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f28f605f826]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x5594fa406fe6]

2020-12-31T23:39:18.097 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.809 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:57.810 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.102176+0000",

2020-12-31T23:39:18.098 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",
, 2020-12-31T07:50:57.831 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:24.500852+0000",

2020-12-31T23:39:18.099 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.850 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "op_commit",
, 2020-12-31T07:50:57.851 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.650960+0000",

2020-12-31T23:39:18.099 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",
, 2020-12-31T07:50:57.871 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:30.575183+0000",

2020-12-31T23:39:18.100 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:                ]
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:            }
, 2020-12-31T07:50:57.892 INFO:teuthology.orchestra.run.smithi093.stdout:        },

2020-12-31T23:39:18.100 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:57.912 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",

2020-12-31T23:39:18.101 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.932 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.933 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",

2020-12-31T23:39:18.102 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.952 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.953 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "started",

2020-12-31T23:39:18.102 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:57.973 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "commit_sent",

2020-12-31T23:39:18.103 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:57.993 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {
, 2020-12-31T07:50:57.994 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",

2020-12-31T23:39:18.103 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 6.2530000000000001e-06
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.014 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:18.104 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.034 INFO:teuthology.orchestra.run.smithi093.stdout:            "type_data": {
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "flag_point": "commit sent; apply or cleanup",
, 2020-12-31T07:50:58.035 INFO:teuthology.orchestra.run.smithi093.stdout:                "client_info": {

2020-12-31T23:39:18.105 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.054 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:49:47.301027+0000",
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.81516294099999997
, 2020-12-31T07:50:58.055 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:18.105 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "queued_for_pg",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.400750+0000",
, 2020-12-31T07:50:58.075 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.2364999999999998e-05

2020-12-31T23:39:18.106 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "all_read",
, 2020-12-31T07:50:58.095 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:47.622931+0000",

2020-12-31T23:39:18.106 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.591 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client": "client.4431",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "client_addr": "172.21.15.93:0/2543268328",
, 2020-12-31T07:50:58.592 INFO:teuthology.orchestra.run.smithi093.stdout:                    "tid": 4001

2020-12-31T23:39:18.107 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.611 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:48:41.525747+0000",

2020-12-31T23:39:18.108 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.630 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:18.108 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.650 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.651 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "waiting for rw locks",

2020-12-31T23:39:18.109 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 7.8399999999999995e-06
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.670 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:18.109 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:        {
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:10069 3.0 3:39cb8851:::smithi09338363-342 low_tier:head [read 0~1] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e77)",
, 2020-12-31T07:50:58.690 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:14.150340+0000",

2020-12-31T23:39:18.110 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:25.319281+0000",
, 2020-12-31T07:50:58.709 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.00012395200000000001
, 2020-12-31T07:50:58.710 INFO:teuthology.orchestra.run.smithi093.stdout:                    },

2020-12-31T23:39:18.111 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "dispatched",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.898369+0000",
, 2020-12-31T07:50:58.729 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 4.9640000000000002e-06

2020-12-31T23:39:18.111 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.749 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "throttled",
, 2020-12-31T07:50:58.750 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.899108+0000",

2020-12-31T23:39:18.112 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "reached_pg",
, 2020-12-31T07:50:58.770 INFO:teuthology.orchestra.run.smithi093.stdout:                        "time": "2020-12-31T07:50:41.906406+0000",

2020-12-31T23:39:18.112 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                    {
, 2020-12-31T07:50:58.790 INFO:teuthology.orchestra.run.smithi093.stdout:                        "event": "header_read",

2020-12-31T23:39:18.113 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.809 INFO:teuthology.orchestra.run.smithi093.stdout:            "description": "osd_op(client.4431.0:12818 3.0 3:1d1630c6:::smithi09338363-401 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [stat,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected e93)",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "initiated_at": "2020-12-31T07:50:52.246307+0000",
, 2020-12-31T07:50:58.810 INFO:teuthology.orchestra.run.smithi093.stdout:            "age": 6.3414430709999996,

2020-12-31T23:39:18.113 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:50:58.830 INFO:teuthology.orchestra.run.smithi093.stdout:                        "duration": 0.97759732200000005
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    },
, 2020-12-31T07:50:58.831 INFO:teuthology.orchestra.run.smithi093.stdout:                    {

2020-12-31T23:39:18.114 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:00.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops
, 2020-12-31T07:51:00.797 INFO:teuthology.orchestra.run.smithi093.stdout:ERROR: (22) Invalid argument
, 2020-12-31T07:51:00.798 INFO:teuthology.orchestra.run.smithi093.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.

2020-12-31T23:39:18.115 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efbd13d3:::smithi09338363-211:head#
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:snapset 0=[]:{}
, 2020-12-31T07:51:08.063 INFO:teuthology.orchestra.run.smithi196.stdout:Write #4:efeb37a5:::smithi09338363-410:head#

2020-12-31T23:39:18.117 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:42.256 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1
, 2020-12-31T07:51:43.248+0000 7f313a583700 -1 osd.0 114 handle_osd_map injecting map crc failure
, set osd.1 primary-affinity to 1 (8655362)

2020-12-31T23:39:18.118 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:51:54.828 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0
, 2020-12-31T07:51:54.979 INFO:teuthology.orchestra.run.smithi093.stdout:{
, 2020-12-31T07:51:54.980 INFO:teuthology.orchestra.run.smithi093.stdout:    "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) "

2020-12-31T23:39:18.119 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 277, in task
,     running.get()

2020-12-31T23:39:18.119 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:52:42.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid
, 2020-12-31T07:52:42.432 INFO:tasks.ceph:Checking for errors in any valgrind logs...

2020-12-31T23:39:18.120 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-31T07:53:59.083 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
, 2020-12-31T07:53:59.111 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core

2020-12-31T23:39:18.145 INFO:scrape:Found 10 distinct failure reasons
2020-12-31T23:39:18.147 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds
2020-12-31T23:39:18.147 INFO:scrape:4 jobs: ['5749005', '5749038', '5748945', '5748907']
2020-12-31T23:39:18.148 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start']
2020-12-31T23:39:18.148 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']
2020-12-31T23:39:18.149 INFO:scrape:
2020-12-31T23:39:18.149 INFO:scrape:Failure: Command failed on smithi037 with status 1: 'sudo docker kill -s 1 ceph-1e2c34cc-4b45-11eb-8f8d-001a4aab830c-osd.10'
2020-12-31T23:39:18.150 INFO:scrape:1 jobs: ['5748845']
2020-12-31T23:39:18.150 INFO:scrape:suites: ['1-install/mimic', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}']
2020-12-31T23:39:18.150 INFO:scrape:
2020-12-31T23:39:18.151 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2020-12-31T23:39:18.151 INFO:scrape:2 jobs: ['5748816', '5748960']
2020-12-31T23:39:18.152 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2020-12-31T23:39:18.152 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2020-12-31T23:39:18.153 INFO:scrape:
2020-12-31T23:39:18.153 INFO:scrape:Dead: b'2020-12-31T21:32:11.994 DEBUG:teuthology.exit:Finished running handlers'
2020-12-31T23:39:18.154 INFO:scrape:4 jobs: ['5748926', '5748998', '5748856', '5749129']
2020-12-31T23:39:18.154 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'openstack}', 'rados', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health']
2020-12-31T23:39:18.155 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-small}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}']
2020-12-31T23:39:18.155 INFO:scrape:
2020-12-31T23:39:18.156 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2020-12-31T23:39:18.156 INFO:scrape:1 jobs: ['5749032']
2020-12-31T23:39:18.157 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']
2020-12-31T23:39:18.157 INFO:scrape:
2020-12-31T23:39:18.157 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
2020-12-31T23:39:18.158 INFO:scrape:1 jobs: ['5748890']
2020-12-31T23:39:18.158 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rhel_8}}']
2020-12-31T23:39:18.159 INFO:scrape:
2020-12-31T23:39:18.159 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2020-12-31T23:39:18.160 INFO:scrape:1 jobs: ['5748847']
2020-12-31T23:39:18.160 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/max-pg-per-osd.from-mon', 'supported-random-distro$/{ubuntu_latest}}']
2020-12-31T23:39:18.161 INFO:scrape:
2020-12-31T23:39:18.161 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'
2020-12-31T23:39:18.161 INFO:scrape:1 jobs: ['5748855']
2020-12-31T23:39:18.162 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}']
2020-12-31T23:39:18.162 INFO:scrape:
2020-12-31T23:39:18.163 INFO:scrape:Failure: Command failed on smithi092 with status 1: 'sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump'
2020-12-31T23:39:18.163 INFO:scrape:2 jobs: ['5748961', '5749029']
2020-12-31T23:39:18.164 INFO:scrape:suites intersection: []
2020-12-31T23:39:18.164 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/singleton/{all/resolve_stuck_peering', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'tasks/dashboard}']
2020-12-31T23:39:18.164 INFO:scrape:
2020-12-31T23:39:18.165 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 50 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'
2020-12-31T23:39:18.165 INFO:scrape:1 jobs: ['5748785']
2020-12-31T23:39:18.166 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect_promote_tests}']
2020-12-31T23:39:18.166 INFO:scrape:
2020-12-31T23:39:18.178 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/
failed:  14
dead:    4
running: 0
waiting: 0
queued:  0
passed:  349


Fail
=================================================================
[5748785]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests}
-----------------------------------------------------------------
time:   00:14:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748785/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748785/

    Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect
    --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16
    --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-
    seconds 0 --op read 50 --op write 50 --op delete 10 --op write_excl 50
    --pool unique_pool_0'

[5748816]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind}
-----------------------------------------------------------------
time:   06:45:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748816/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748816/
sentry: https://sentry.ceph.com/organizations/ceph/?query=155349158c9e4cac9bdd0d9ce8a3f493

    Command failed (workunit test rados/test.sh) on smithi089 with status 124:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

[5748845]  rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/cache-snaps}
-----------------------------------------------------------------
time:   00:27:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748845/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748845/

    Command failed on smithi037 with status 1: 'sudo docker kill -s 1
    ceph-1e2c34cc-4b45-11eb-8f8d-001a4aab830c-osd.10'

[5748847]  rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:07:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748847/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748847/

    Command failed on smithi110 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[5748855]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption}
-----------------------------------------------------------------
time:   00:14:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748855/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748855/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8a3b149db81a4763b17eb5a2eb2d5aaa

    Command failed (workunit test cephadm/test_adoption.sh) on smithi181 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'

[5748890]  rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}}
-----------------------------------------------------------------
time:   00:33:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748890/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748890/
sentry: https://sentry.ceph.com/organizations/ceph/?query=39a94036092b4a1cb696a6f5b59d4821

    Command failed (workunit test cephtool/test.sh) on smithi133 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

[5748907]  rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic}
-----------------------------------------------------------------
time:   00:16:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748907/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748907/
sentry: https://sentry.ceph.com/organizations/ceph/?query=968f19ae365f4866835d5b750fce0822

    reached maximum tries (180) after waiting for 180 seconds

[5748945]  rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity}
-----------------------------------------------------------------
time:   00:17:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748945/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748945/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f4772d497b6418488487f61a14671b3

    reached maximum tries (180) after waiting for 180 seconds

[5748960]  rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}
-----------------------------------------------------------------
time:   06:47:03
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748960/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748960/
sentry: https://sentry.ceph.com/organizations/ceph/?query=11fa476a832a46fbb3405c575116a833

    Command failed (workunit test rados/test.sh) on smithi135 with status 124:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-
    ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

[5748961]  rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard}
-----------------------------------------------------------------
time:   01:09:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748961/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748961/

    Command failed on smithi092 with status 1: 'sudo sysctl -w
    kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty --
    /home/ubuntu/cephtest/archive/coredump'

[5749005]  rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic}
-----------------------------------------------------------------
time:   00:19:03
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749005/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749005/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d081d64c1c7844ffb2952eb44930a2ec

    reached maximum tries (180) after waiting for 180 seconds

[5749029]  rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:09:21
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749029/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749029/

    Command failed on smithi164 with status 1: 'sudo sysctl -w
    kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty --
    /home/ubuntu/cephtest/archive/coredump'

[5749032]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm}
-----------------------------------------------------------------
time:   00:14:25
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749032/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749032/
sentry: https://sentry.ceph.com/organizations/ceph/?query=58bdb2aa1820422981be23b1e43d9b97

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi007 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=1bfac6181a18be3417e8090e0e2894c39783abc1
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[5749038]  rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity}
-----------------------------------------------------------------
time:   00:17:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749038/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749038/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f9dfb6d92509430bb42e5ffdf00b6991

    reached maximum tries (180) after waiting for 180 seconds



Dead
=================================================================
[5748856]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748856/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748856/

[5748926]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-small}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748926/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748926/

[5748998]  rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748998/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748998/

[5749129]  rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749129/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749129/



Pass
=================================================================
[5748786] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews}
time:   00:05:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748786/

[5748787] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:26:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748787/

[5748788] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748788/

[5748789] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read}
time:   00:08:08
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748789/

[5748790] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}
time:   00:30:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748790/

[5748791] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests}
time:   00:40:21
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748791/

[5748792] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
time:   00:34:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748792/

[5748793] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start}
time:   00:11:27
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748793/

[5748794] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/rados_striper}
time:   00:09:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748794/

[5748795] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748795/

[5748796] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/crash}
time:   00:08:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748796/

[5748797] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}}
time:   00:46:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748797/

[5748798] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:08:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748798/

[5748799] rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}}
time:   00:12:56
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748799/

[5748800] rados/rest/{mgr-restful supported-random-distro$/{centos_8}}
time:   00:17:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748800/

[5748801] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:15:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748801/

[5748802] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/crush}
time:   00:19:16
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748802/

[5748803] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest}
time:   05:06:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748803/

[5748804] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest}
time:   00:20:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748804/

[5748805] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects}
time:   00:34:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748805/

[5748806] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object}
time:   00:12:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748806/

[5748807] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e}
time:   00:32:37
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748807/

[5748808] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}
time:   00:44:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748808/

[5748809] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}}
time:   00:12:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748809/

[5748810] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:17:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748810/

[5748811] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start}
time:   00:13:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748811/

[5748812] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/set-chunks-read}
time:   00:14:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748812/

[5748813] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw}
time:   00:08:27
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748813/

[5748814] rados/cephadm/smoke-roleless/{distro/centos_8.0 start}
time:   00:13:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748814/

[5748815] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:09:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748815/

[5748817] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:28:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748817/

[5748818] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}}
time:   00:15:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748818/

[5748819] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:08:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748819/

[5748820] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/classic}
time:   00:16:24
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748820/

[5748821] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:09:03
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748821/

[5748822] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python}
time:   00:27:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748822/

[5748823] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced}
time:   00:19:08
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748823/

[5748824] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/erasure-code}
time:   00:40:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748824/

[5748825] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:11:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748825/

[5748826] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli}
time:   00:19:16
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748826/

[5748827] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews}
time:   00:08:24
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748827/

[5748828] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
time:   00:26:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748828/

[5748829] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big}
time:   00:36:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748829/

[5748830] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}}
time:   00:52:21
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748830/

[5748831] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_mon_osdmap_prune}
time:   01:12:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748831/

[5748832] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start}
time:   00:12:42
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748832/

[5748833] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:17:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748833/

[5748834] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/failover}
time:   00:10:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748834/

[5748835] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api}
time:   00:17:58
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748835/

[5748836] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-localized}
time:   00:18:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748836/

[5748837] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_write}
time:   00:12:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748837/

[5748838] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start}
time:   00:14:59
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748838/

[5748839] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}}
time:   00:53:28
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748839/

[5748840] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:08:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748840/

[5748841] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}}
time:   00:09:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748841/

[5748842] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:15:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748842/

[5748843] rados/cephadm/smoke-roleless/{distro/centos_latest start}
time:   00:12:11
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748843/

[5748844] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
time:   00:23:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748844/

[5748846] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects}
time:   00:25:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748846/

[5748848] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:39:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748848/

[5748849] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr}
time:   00:19:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748849/

[5748850] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:26:59
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748850/

[5748851] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/lockdep}
time:   00:26:11
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748851/

[5748852] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:08:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748852/

[5748853] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:21:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748853/

[5748854] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start}
time:   00:12:13
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748854/

[5748857] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix}
time:   00:25:43
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748857/

[5748858] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
time:   00:13:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748858/

[5748859] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:07:36
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748859/

[5748860] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748860/

[5748861] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read}
time:   00:10:16
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748861/

[5748862] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/connectivity}
time:   00:16:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748862/

[5748863] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_recovery}
time:   00:11:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748863/

[5748864] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
time:   00:24:43
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748864/

[5748865] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:05:25
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748865/

[5748866] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start}
time:   00:11:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748866/

[5748867] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits}
time:   00:36:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748867/

[5748868] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start}
time:   00:14:13
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748868/

[5748869] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/insights}
time:   00:13:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748869/

[5748870] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}
time:   00:09:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748870/

[5748871] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized}
time:   00:26:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748871/

[5748872] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/radosbench}
time:   01:19:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748872/

[5748873] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:17:54
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748873/

[5748874] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python}
time:   00:29:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748874/

[5748875] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/misc}
time:   00:31:59
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748875/

[5748876] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}}
time:   00:17:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748876/

[5748877] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start}
time:   00:12:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748877/

[5748878] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:11:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748878/

[5748879] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}}
time:   00:08:30
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748879/

[5748880] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects}
time:   00:24:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748880/

[5748881] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read}
time:   00:08:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748881/

[5748882] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/rbd_cls}
time:   00:15:11
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748882/

[5748883] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:16:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748883/

[5748884] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start}
time:   00:11:43
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748884/

[5748885] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:11:52
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748885/

[5748886] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind}
time:   00:26:34
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748886/

[5748887] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread}
time:   00:39:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748887/

[5748888] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:25:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748888/

[5748889] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}}
time:   00:19:15
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748889/

[5748891] rados/cephadm/smoke-roleless/{distro/rhel_latest start}
time:   00:11:58
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748891/

[5748892] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed}
time:   00:16:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748892/

[5748893] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:09
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748893/

[5748894] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm}
time:   00:22:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748894/

[5748895] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748895/

[5748896] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-overwrites}
time:   00:23:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748896/

[5748897] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests}
time:   00:40:09
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748897/

[5748898] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:18:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748898/

[5748899] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews}
time:   00:08:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748899/

[5748900] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-radosbench}
time:   02:48:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748900/

[5748901] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:08:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748901/

[5748902] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/snaps-few-objects}
time:   00:26:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748902/

[5748903] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start}
time:   00:13:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748903/

[5748904] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/module_selftest}
time:   00:28:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748904/

[5748905] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon}
time:   00:49:36
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748905/

[5748906] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests}
time:   00:25:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748906/

[5748908] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read}
time:   00:09:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748908/

[5748909] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:17:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748909/

[5748910] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects}
time:   00:36:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748910/

[5748911] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}}
time:   00:17:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748911/

[5748912] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start}
time:   00:12:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748912/

[5748913] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big}
time:   00:30:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748913/

[5748914] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start}
time:   00:10:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748914/

[5748915] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:14:54
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748915/

[5748916] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/readwrite}
time:   00:12:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748916/

[5748917] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}}
time:   00:20:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748917/

[5748918] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:11:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748918/

[5748919] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}}
time:   02:14:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748919/

[5748920] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest}
time:   00:19:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748920/

[5748921] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python}
time:   00:31:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748921/

[5748922] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos}
time:   00:10:57
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748922/

[5748923] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:26:30
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748923/

[5748924] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep}
time:   00:33:11
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748924/

[5748925] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:26:13
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748925/

[5748927] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:11
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748927/

[5748928] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:22:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748928/

[5748929] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/test_rbd_api}
time:   00:19:09
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748929/

[5748930] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read}
time:   00:08:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748930/

[5748931] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd}
time:   03:14:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748931/

[5748932] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}}
time:   00:17:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748932/

[5748933] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start}
time:   00:14:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748933/

[5748934] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start}
time:   00:12:11
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748934/

[5748935] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews}
time:   00:08:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748935/

[5748936] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced}
time:   00:21:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748936/

[5748937] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy}
time:   00:23:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748937/

[5748938] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/progress}
time:   00:24:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748938/

[5748939] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/sync workloads/pool-create-delete}
time:   00:14:41
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748939/

[5748940] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:18:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748940/

[5748941] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}
time:   00:25:46
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748941/

[5748942] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}}
time:   00:14:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748942/

[5748943] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:10:56
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748943/

[5748944] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/cache-snaps}
time:   00:30:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748944/

[5748946] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps}
time:   00:19:58
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748946/

[5748947] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites}
time:   00:26:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748947/

[5748948] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:13:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748948/

[5748949] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/repair_test}
time:   00:13:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748949/

[5748950] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
time:   00:09:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748950/

[5748951] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:40:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748951/

[5748952] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write}
time:   00:07:57
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748952/

[5748953] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start}
time:   00:11:53
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748953/

[5748954] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/scrub}
time:   01:42:53
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748954/

[5748955] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli}
time:   00:19:16
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748955/

[5748956] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:27:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748956/

[5748957] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
time:   00:19:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748957/

[5748958] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced}
time:   00:23:56
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748958/

[5748959] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:08:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748959/

[5748962] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{rhel_8}}
time:   02:14:35
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748962/

[5748963] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:37:27
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748963/

[5748964] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:15:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748964/

[5748965] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/radosbench}
time:   01:38:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748965/

[5748966] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start}
time:   00:12:13
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748966/

[5748967] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start}
time:   00:10:57
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748967/

[5748968] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
time:   00:16:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748968/

[5748969] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:17:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748969/

[5748970] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-snaps}
time:   00:23:35
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748970/

[5748971] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_recovery}
time:   00:10:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748971/

[5748972] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read}
time:   00:17:36
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748972/

[5748973] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/prometheus}
time:   00:12:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748973/

[5748974] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:27:08
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748974/

[5748975] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_omap_write}
time:   00:20:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748975/

[5748976] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_5925}
time:   00:11:52
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748976/

[5748977] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start}
time:   00:11:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748977/

[5748978] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test}
time:   00:11:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748978/

[5748979] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
time:   00:08:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748979/

[5748980] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/rbd_cls}
time:   00:18:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748980/

[5748981] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
time:   00:43:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748981/

[5748982] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:07:37
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748982/

[5748983] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:16:54
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748983/

[5748984] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush}
time:   00:15:16
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748984/

[5748985] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest}
time:   05:07:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748985/

[5748986] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache}
time:   00:13:37
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748986/

[5748987] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption}
time:   00:16:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748987/

[5748988] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e}
time:   00:33:48
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748988/

[5748989] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:19:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748989/

[5748990] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}}
time:   00:13:09
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748990/

[5748991] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:17:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748991/

[5748992] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects}
time:   00:33:03
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748992/

[5748993] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748993/

[5748994] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start}
time:   00:14:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748994/

[5748995] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_fio}
time:   00:11:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748995/

[5748996] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:26:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748996/

[5748997] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep}
time:   00:25:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748997/

[5748999] rados/cephadm/smoke-roleless/{distro/centos_8.0 start}
time:   00:14:35
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5748999/

[5749000] rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}}
time:   00:33:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749000/

[5749001] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
time:   00:25:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749001/

[5749002] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:09:30
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749002/

[5749003] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}}
time:   00:15:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749003/

[5749004] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:09:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749004/

[5749006] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests}
time:   00:33:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749006/

[5749007] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
time:   00:40:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749007/

[5749008] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/workunits}
time:   00:13:46
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749008/

[5749009] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/erasure-code}
time:   00:41:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749009/

[5749010] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews}
time:   00:08:34
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749010/

[5749011] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-many-deletes}
time:   00:27:57
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749011/

[5749012] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_api_tests}
time:   00:33:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749012/

[5749013] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/snaps-few-objects}
time:   00:29:08
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749013/

[5749014] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_api_tests}
time:   00:38:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749014/

[5749015] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:15:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749015/

[5749016] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}}
time:   00:16:48
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749016/

[5749017] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start}
time:   00:12:59
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749017/

[5749018] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency}
time:   00:53:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749018/

[5749019] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start}
time:   00:14:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749019/

[5749020] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench}
time:   00:08:25
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749020/

[5749021] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
time:   00:16:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749021/

[5749022] rados/cephadm/smoke-roleless/{distro/centos_latest start}
time:   00:12:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749022/

[5749023] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:07:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749023/

[5749024] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:15:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749024/

[5749025] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python}
time:   00:29:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749025/

[5749026] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench}
time:   02:03:27
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749026/

[5749027] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749027/

[5749028] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api}
time:   00:18:46
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749028/

[5749030] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:13:13
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749030/

[5749031] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind}
time:   00:28:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749031/

[5749033] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:31:58
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749033/

[5749034] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mgr}
time:   00:19:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749034/

[5749035] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:04:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749035/

[5749036] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start}
time:   00:12:27
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749036/

[5749037] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect}
time:   00:12:54
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749037/

[5749039] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_cls_all}
time:   00:24:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749039/

[5749040] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
time:   00:17:41
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749040/

[5749041] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write}
time:   00:19:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749041/

[5749042] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/crash}
time:   00:11:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749042/

[5749043] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:08:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749043/

[5749044] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}}
time:   00:11:51
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749044/

[5749045] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest}
time:   00:21:02
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749045/

[5749046] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps}
time:   00:29:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749046/

[5749047] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews}
time:   00:08:21
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749047/

[5749048] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects}
time:   00:19:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749048/

[5749049] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:05:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749049/

[5749050] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_osdmap_prune}
time:   01:07:50
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749050/

[5749051] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start}
time:   00:11:38
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749051/

[5749052] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_promote_tests}
time:   00:13:35
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749052/

[5749053] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
time:   00:24:28
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749053/

[5749054] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start}
time:   00:14:50
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749054/

[5749055] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
time:   00:21:41
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749055/

[5749056] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests}
time:   00:39:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749056/

[5749057] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:14:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749057/

[5749058] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start}
time:   00:12:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749058/

[5749059] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/misc}
time:   00:35:50
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749059/

[5749060] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}}
time:   01:19:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749060/

[5749061] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object}
time:   00:15:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749061/

[5749062] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:07:58
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749062/

[5749063] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:08:48
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749063/

[5749064] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos}
time:   00:13:34
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749064/

[5749065] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/cosbench_64K_write}
time:   00:21:20
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749065/

[5749066] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench}
time:   01:21:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749066/

[5749067] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}}
time:   00:17:08
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749067/

[5749068] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:28:25
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749068/

[5749069] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}
time:   00:32:52
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749069/

[5749070] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:23:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749070/

[5749071] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start}
time:   00:11:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749071/

[5749072] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_python}
time:   00:19:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749072/

[5749073] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
time:   00:51:45
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749073/

[5749074] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}}
time:   00:45:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749074/

[5749075] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read}
time:   00:16:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749075/

[5749076] rados/cephadm/smoke-roleless/{distro/rhel_latest start}
time:   00:11:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749076/

[5749077] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/failover}
time:   00:14:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749077/

[5749078] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python}
time:   00:32:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749078/

[5749079] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:09:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749079/

[5749080] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_recovery}
time:   00:20:59
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749080/

[5749081] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:28:37
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749081/

[5749082] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}
time:   00:28:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749082/

[5749083] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls}
time:   00:22:48
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749083/

[5749084] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:08:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749084/

[5749085] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}}
time:   00:31:15
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749085/

[5749086] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/rados_mon_workunits}
time:   00:36:24
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749086/

[5749087] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-balanced}
time:   00:20:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749087/

[5749088] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start}
time:   00:13:28
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749088/

[5749089] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mon}
time:   00:44:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749089/

[5749090] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read}
time:   00:11:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749090/

[5749091] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:17:22
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749091/

[5749092] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic}
time:   00:18:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749092/

[5749093] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}}
time:   00:34:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749093/

[5749094] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli}
time:   00:17:30
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749094/

[5749095] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized}
time:   00:26:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749095/

[5749096] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:24
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749096/

[5749097] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start}
time:   00:12:52
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749097/

[5749098] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:14:13
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749098/

[5749099] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start}
time:   00:11:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749099/

[5749100] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch}
time:   00:24:56
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749100/

[5749101] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}}
time:   00:09:23
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749101/

[5749102] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard}
time:   01:09:55
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749102/

[5749103] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
time:   00:40:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749103/

[5749104] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:09:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749104/

[5749105] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind}
time:   02:15:46
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749105/

[5749106] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-overwrites}
time:   00:21:10
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749106/

[5749107] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:30:40
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749107/

[5749108] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects}
time:   00:25:21
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749108/

[5749109] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}
time:   00:39:26
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749109/

[5749110] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw}
time:   00:11:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749110/

[5749111] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
time:   00:18:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749111/

[5749112] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/insights}
time:   00:13:37
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749112/

[5749113] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start}
time:   00:15:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749113/

[5749114] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/osd}
time:   03:14:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749114/

[5749115] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
time:   00:05:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749115/

[5749116] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start}
time:   00:12:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749116/

[5749117] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews}
time:   00:05:04
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749117/

[5749118] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
time:   00:26:37
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749118/

[5749119] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}
time:   00:30:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749119/

[5749120] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}
time:   00:19:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749120/

[5749121] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/sync workloads/snaps-few-objects}
time:   00:31:18
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749121/

[5749122] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
time:   00:19:15
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749122/

[5749123] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}
time:   00:12:44
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749123/

[5749124] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api}
time:   00:14:19
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749124/

[5749125] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:11:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749125/

[5749126] rados/objectstore/{backends/objectstore supported-random-distro$/{ubuntu_latest}}
time:   05:17:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749126/

[5749127] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python}
time:   00:27:08
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749127/

[5749128] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:08:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749128/

[5749130] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption}
time:   00:16:05
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749130/

[5749131] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}}
time:   00:16:29
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749131/

[5749132] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_striper}
time:   00:10:09
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749132/

[5749133] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}}
time:   00:11:07
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749133/

[5749134] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read}
time:   00:08:01
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749134/

[5749135] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/connectivity}
time:   00:17:17
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749135/

[5749136] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/scrub}
time:   01:47:33
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749136/

[5749137] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start}
time:   00:11:35
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749137/

[5749138] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:27:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749138/

[5749139] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep}
time:   00:12:35
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749139/

[5749140] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:27:14
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749140/

[5749141] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}
time:   00:23:47
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749141/

[5749142] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps}
time:   00:29:56
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749142/

[5749143] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}}
time:   00:29:31
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749143/

[5749144] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:45
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749144/

[5749145] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard}
time:   01:08:06
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749145/

[5749146] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start}
time:   00:12:12
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749146/

[5749147] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/module_selftest}
time:   00:25:32
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749147/

[5749148] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start}
time:   00:11:00
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749148/

[5749149] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:39
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749149/

[5749150] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests}
time:   00:39:49
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749150/

[5749151] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed}
time:   00:14:21
info:   http://pulpito.ceph.com/teuthology-2020-12-31_07:01:02-rados-master-distro-basic-smithi/5749151/