2020-10-30T04:41:31.800 INFO:root:teuthology version: 1.0.0-f7de558 2020-10-30T04:41:31.802 INFO:teuthology.results:subset: '918/9999' 2020-10-30T04:41:31.802 INFO:teuthology.results:seed: '2477' 2020-10-30T10:19:28.141 INFO:root:teuthology version: 1.0.0-f7de558 2020-10-30T10:19:28.202 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-30T17:44:46.560 INFO:teuthology.results:Tests finished! gathering results... 2020-10-30T17:45:53.646 INFO:scrape:Found 446 jobs 2020-10-30T17:46:17.935 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:17.936 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:17.938 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:17.938 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:17.939 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:17.939 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.039 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.040 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.043 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.150 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.151 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.153 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.153 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.154 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.155 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.257 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.258 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.260 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.260 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.261 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.261 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.360 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.361 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.362 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.363 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.364 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.364 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.466 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.467 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.469 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.469 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.470 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.470 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.574 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.575 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.577 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.577 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.578 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.578 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.699 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.700 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.702 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.702 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.703 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.703 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.804 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.805 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.807 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.807 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.808 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.808 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:18.937 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:18.938 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:18.939 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:18.940 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:18.940 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:18.941 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:19.039 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:19.040 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:19.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:19.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:19.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:19.043 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:19.146 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:19.147 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:19.148 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:19.149 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:19.149 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:19.150 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:19.267 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f9fdd7bb9b4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x563066bc9ca6] 2020-10-30T17:46:19.268 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.094+0000 7f5d02bcc700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:08.117 INFO:teuthology.orchestra.run.smithi180.stdout:{ 2020-10-30T17:46:19.269 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:05:10.660 INFO:teuthology.orchestra.run.smithi180.stdout: , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-10-30T06:05:10.658+0000 7fc2906f2700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-10-30T17:46:19.270 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:06:12.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-30T06:06:12.043 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-30T06:06:12.076 INFO:tasks.ceph.osd.0:Stopped 2020-10-30T17:46:19.270 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-30T06:08:24.893 INFO:teuthology.orchestra.run.smithi180:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found , 2020-10-30T06:08:24.912 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-10-30T17:46:19.271 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 172, in run_tasks 2020-10-30T17:46:25.253 INFO:scrape:Found 14 distinct failure reasons 2020-10-30T17:46:25.254 INFO:scrape:Crash: Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' 2020-10-30T17:46:25.255 INFO:scrape:ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f41522e98a0] 2: gsignal() 3: abort() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x3042a) [0x7f41513d342a] 5: /lib/x86_64-linux-gnu/libc.so.6(+0x304a2) [0x7f41513d34a2] 6: /usr/lib/ceph/libceph-common.so.2(+0x41fa36) [0x7f4153230a36] 7: (AsyncMessenger::shutdown_connections(bool)+0x1f3) [0x7f415330ab23] 8: (AsyncMessenger::shutdown()+0x86) [0x7f415330b1a6] 9: /usr/lib/ceph/libceph-common.so.2(+0x5a18ea) [0x7f41533b28ea] 10: (MonClient::get_monmap_and_config()+0x3e5) [0x7f41533bd695] 11: (global_init(std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const*, std::vector >&, unsigned int, code_environment_t, int, bool)+0x16b5) [0x55c951eb1dc5] 12: main() 13: __libc_start_main() 14: _start() 2020-10-30T17:46:25.255 INFO:scrape:76 jobs 2020-10-30T17:46:25.256 INFO:scrape:suites intersection: [] 2020-10-30T17:46:25.256 INFO:scrape:suites union: ['1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'debug/mgr', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack', 'openstack}', 'orchestrator_cli', 'rados', 'rados/basic/{ceph', 'rados/cephadm/orchestrator_cli/{2-node-mgr', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/monthrash/{ceph', 'rados/objectstore/{backends/ceph_objectstore_tool', 'rados/objectstore/{backends/filestore-idempotent', 'rados/objectstore/{backends/objectcacher-stress', 'rados/perf/{ceph', 'rados/singleton/{all/thrash-rados/{thrash-rados', 'rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'settings/optimized', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/crash}', 'tasks/dashboard}', 'tasks/prometheus}', 'tasks/rados_cls_all}', 'tasks/rados_python}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashers/sync-many', 'thrashosds-health', 'thrashosds-health}', 'ubuntu_latest', 'ubuntu_latest}', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}', 'workloads/crush}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-many-deletes}', 'workloads/erasure-code}', 'workloads/fio_4K_rand_read}', 'workloads/fio_4K_rand_rw}', 'workloads/fio_4M_rand_read}', 'workloads/fio_4M_rand_write}', 'workloads/osd}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench_4K_seq_read}', 'workloads/radosbench_4M_write}', 'workloads/radosbench_omap_write}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/sample_fio}', 'workloads/scrub}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] 2020-10-30T17:46:25.257 INFO:scrape: 2020-10-30T17:46:25.257 INFO:scrape:Failure: Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' 2020-10-30T17:46:25.257 INFO:scrape:39 jobs 2020-10-30T17:46:25.258 INFO:scrape:suites intersection: [] 2020-10-30T17:46:25.258 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'cluster/{12-osds', 'clusters/9-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/default', 'd-balancer/crush-compat', 'fast/fast', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack', 'openstack}', 'rados', 'rados/monthrash/{ceph', 'rados/multimon/{clusters/3', 'rados/multimon/{clusters/6', 'rados/multimon/{clusters/9', 'rados/objectstore/{backends/alloc-hint', 'rados/perf/{ceph', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/singleton-nomsgr/{all/cache-fs-trunc', 'rados/singleton-nomsgr/{all/export-after-evict', 'rados/singleton-nomsgr/{all/health-warnings', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/singleton-nomsgr/{all/librados_hello_world', 'rados/singleton-nomsgr/{all/osd_stale_reads', 'rados/singleton-nomsgr/{all/pool-access', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/divergent_priors2', 'rados/singleton/{all/lost-unfound', 'rados/singleton/{all/lost-unfound-delete', 'rados/singleton/{all/max-pg-per-osd.from-primary', 'rados/singleton/{all/max-pg-per-osd.from-replica', 'rados/singleton/{all/mon-config', 'rados/singleton/{all/peer', 'rados/singleton/{all/pg-removal-interruption', 'rados/singleton/{all/radostool', 'rados/singleton/{all/rebuild-mondb', 'rados/singleton/{all/recovery-preemption', 'rados/singleton/{all/test-crash', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'settings/optimized', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/mon_clock_no_skews}', 'tasks/mon_recovery}', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/pggrow', 'thrashers/sync', 'thrashosds-health', 'ubuntu_latest', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects}', 'workloads/fio_4M_rand_rw}', 'workloads/rados_mon_osdmap_prune}', 'workloads/radosbench_4K_rand_read}', 'workloads/radosbench_4M_rand_read}', 'workloads/radosbench_4M_seq_read}', 'workloads/sample_radosbench}', 'workloads/snaps-few-objects}'] 2020-10-30T17:46:25.259 INFO:scrape: 2020-10-30T17:46:25.259 INFO:scrape:Failure: Command failed on smithi140 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' 2020-10-30T17:46:25.259 INFO:scrape:1 jobs: ['5573213'] 2020-10-30T17:46:25.260 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] 2020-10-30T17:46:25.260 INFO:scrape: 2020-10-30T17:46:25.260 INFO:scrape:Crash: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_msgr' 2020-10-30T17:46:25.261 INFO:scrape:ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7fc5167ec8a0] 2: gsignal() 3: abort() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x3042a) [0x7fc50bb1542a] 5: /lib/x86_64-linux-gnu/libc.so.6(+0x304a2) [0x7fc50bb154a2] 6: /usr/lib/ceph/libceph-common.so.2(+0x41fa36) [0x7fc50d040a36] 7: (AsyncConnection::_stop()+0xd9) [0x7fc50d10fba9] 8: (ProtocolV2::stop()+0x8b) [0x7fc50d146b9b] 9: (AsyncConnection::stop(bool)+0x6f) [0x7fc50d11021f] 10: (AsyncMessenger::shutdown_connections(bool)+0x209) [0x7fc50d11ab39] 11: (AsyncMessenger::shutdown()+0x86) [0x7fc50d11b1a6] 12: (MessengerTest_ConnectionRaceTest_Test::TestBody()+0xcb8) [0x559ccce7a898] 13: (void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*)+0x4a) [0x559cccee3ada] 14: (testing::Test::Run()+0xc3) [0x559ccced8f93] 15: (testing::TestInfo::Run()+0x135) [0x559ccced90f5] 16: (testing::TestSuite::Run()+0xb5) [0x559ccced91d5] 17: (testing::internal::UnitTestImpl::RunAllTests()+0x478) [0x559ccced96e8] 18: (bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x4a) [0x559cccee3fea] 19: (testing::UnitTest::Run()+0x98) [0x559ccced98f8] 20: main() 21: __libc_start_main() 22: _start() 2020-10-30T17:46:25.261 INFO:scrape:1 jobs: ['5573006'] 2020-10-30T17:46:25.261 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/msgr', 'supported-random-distro$/{ubuntu_latest}}'] 2020-10-30T17:46:25.262 INFO:scrape: 2020-10-30T17:46:25.262 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2020-10-30T17:46:25.262 INFO:scrape:4 jobs: ['5573430', '5573237', '5573264', '5573457'] 2020-10-30T17:46:25.263 INFO:scrape:suites intersection: ['mon_election/connectivity'] 2020-10-30T17:46:25.263 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/cephadm/with-work/{distro/ubuntu_18.04', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'start', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/rados_api_tests}', 'thrashers/careful', 'thrashers/mapgap', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-10-30T17:46:25.263 INFO:scrape: 2020-10-30T17:46:25.264 INFO:scrape:Failure: Command failed (workunit test rados/test_python.sh) on smithi103 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' 2020-10-30T17:46:25.264 INFO:scrape:3 jobs: ['5572988', '5573456', '5573266'] 2020-10-30T17:46:25.264 INFO:scrape:suites intersection: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'rados/cephadm/with-work/{distro/ubuntu_18.04_podman', 'start', 'tasks/rados_python}'] 2020-10-30T17:46:25.264 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'rados/cephadm/with-work/{distro/ubuntu_18.04_podman', 'start', 'tasks/rados_python}'] 2020-10-30T17:46:25.265 INFO:scrape: 2020-10-30T17:46:25.265 INFO:scrape:Crash: Command crashed: 'sudo ceph --cluster ceph osd crush tunables jewel' 2020-10-30T17:46:25.265 INFO:scrape:ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7fc53ced68a0] 2: gsignal() 3: abort() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x3042a) [0x7fc53bfc042a] 5: /lib/x86_64-linux-gnu/libc.so.6(+0x304a2) [0x7fc53bfc04a2] 6: /usr/lib/ceph/libceph-common.so.2(+0x41fa36) [0x7fc53de1da36] 7: (AsyncConnection::_stop()+0xd9) [0x7fc53deecba9] 8: (ProtocolV1::stop()+0xa3) [0x7fc53df177b3] 9: (AsyncConnection::mark_down()+0x66) [0x7fc53def1806] 10: (AsyncMessenger::shutdown()+0xf3) [0x7fc53def8213] 11: /usr/lib/ceph/libceph-common.so.2(+0x5a18ea) [0x7fc53df9f8ea] 12: (MonClient::get_monmap_and_config()+0x3e5) [0x7fc53dfaa695] 13: (global_init(std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const*, std::vector >&, unsigned int, code_environment_t, int, bool)+0x16b5) [0x56436bf1adc5] 14: main() 15: __libc_start_main() 16: _start() 2020-10-30T17:46:25.266 INFO:scrape:1 jobs: ['5573146'] 2020-10-30T17:46:25.266 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-10-30T17:46:25.266 INFO:scrape: 2020-10-30T17:46:25.267 INFO:scrape:Failure: "2020-10-30T06:06:31.063518+0000 osd.7 (osd.7) 10 : cluster [ERR] osd.7 found snap mapper error on pg 3.2e oid 3:7664ebeb:::smithi01437199-697:10f snaps missing in mapper, should be: 10f was r -2...repaired" in cluster log 2020-10-30T17:46:25.267 INFO:scrape:1 jobs: ['5573206'] 2020-10-30T17:46:25.267 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/small-objects-localized}'] 2020-10-30T17:46:25.267 INFO:scrape: 2020-10-30T17:46:25.268 INFO:scrape:Failure: Command failed on smithi142 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' 2020-10-30T17:46:25.268 INFO:scrape:4 jobs: ['5573191', '5573412', '5573310', '5573487'] 2020-10-30T17:46:25.268 INFO:scrape:suites intersection: [] 2020-10-30T17:46:25.269 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'rados/singleton-nomsgr/{all/balancer', 'supported-random-distro$/{centos_8}}', 'task/test_orch_cli}'] 2020-10-30T17:46:25.269 INFO:scrape: 2020-10-30T17:46:25.269 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 2020-10-30T17:46:25.270 INFO:scrape:1 jobs: ['5573227'] 2020-10-30T17:46:25.270 INFO:scrape:suites: ['clusters/3-mons', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{rhel_8}', 'thrashers/force-sync-many', 'workloads/rados_mon_workunits}'] 2020-10-30T17:46:25.270 INFO:scrape: 2020-10-30T17:46:25.270 INFO:scrape:Crash: Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' 2020-10-30T17:46:25.271 INFO:scrape:ceph version 16.0.0-6749-g874c7de6 (874c7de67fc4e3c0fd6168d3999a388dd662b910) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7fe4055538a0] 2: gsignal() 3: abort() 4: /lib/x86_64-linux-gnu/libc.so.6(+0x3042a) [0x7fe40463d42a] 5: /lib/x86_64-linux-gnu/libc.so.6(+0x304a2) [0x7fe40463d4a2] 6: /usr/lib/ceph/libceph-common.so.2(+0x41fa36) [0x7fe40649aa36] 7: (AsyncConnection::_stop()+0xd9) [0x7fe406569ba9] 8: (ProtocolV1::stop()+0xa3) [0x7fe4065947b3] 9: (AsyncConnection::mark_down()+0x66) [0x7fe40656e806] 10: (AsyncMessenger::shutdown()+0xf3) [0x7fe406575213] 11: /usr/lib/ceph/libceph-common.so.2(+0x5a18ea) [0x7fe40661c8ea] 12: (MonClient::get_monmap_and_config()+0x3e5) [0x7fe406627695] 13: (global_init(std::map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const*, std::vector >&, unsigned int, code_environment_t, int, bool)+0x16b5) [0x558242317dc5] 14: main() 15: __libc_start_main() 16: _start() 2020-10-30T17:46:25.271 INFO:scrape:1 jobs: ['5573348'] 2020-10-30T17:46:25.271 INFO:scrape:suites: ['debug/mgr', 'mon_election/connectivity', 'objectstore/bluestore-stupid', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/dashboard}'] 2020-10-30T17:46:25.272 INFO:scrape: 2020-10-30T17:46:25.272 INFO:scrape:Dead: b'2020-10-30T17:42:26.962 DEBUG:teuthology.exit:Finished running handlers' 2020-10-30T17:46:25.272 INFO:scrape:1 jobs: ['5573215'] 2020-10-30T17:46:25.273 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/thrash-eio', 'supported-random-distro$/{rhel_8}}'] 2020-10-30T17:46:25.273 INFO:scrape: 2020-10-30T17:46:25.273 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-10-30T17:46:25.273 INFO:scrape:1 jobs: ['5573238'] 2020-10-30T17:46:25.274 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{centos_8}}'] 2020-10-30T17:46:25.274 INFO:scrape: 2020-10-30T17:46:25.274 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2020-10-30T17:46:25.275 INFO:scrape:1 jobs: ['5573178'] 2020-10-30T17:46:25.275 INFO:scrape:suites: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] 2020-10-30T17:46:25.275 INFO:scrape: 2020-10-30T17:46:25.295 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/ failed: 134 dead: 1 running: 0 waiting: 0 queued: 0 passed: 311 Fail ================================================================= [5572986] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572986/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572986/ sentry: https://sentry.ceph.com/ceph/sepia/?query=f817e340c7794bd08302ced9f4e82284 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5572988] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:16:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572988/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572988/ sentry: https://sentry.ceph.com/ceph/sepia/?query=ac39daf1c4944fac94e34f649b89bba3 Command failed (workunit test rados/test_python.sh) on smithi103 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' [5572990] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:05:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572990/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572990/ sentry: https://sentry.ceph.com/ceph/sepia/?query=8cf98abb4fd04c78b58fbde658ccd9da Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5572996] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:05:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572996/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572996/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9689bf66050345ff8a4d3fc049259039 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573000] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:05:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573000/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573000/ sentry: https://sentry.ceph.com/ceph/sepia/?query=087541315bb64279aa2d47fbfa844f89 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573002] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_omap_write} ----------------------------------------------------------------- time: 00:04:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573002/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573002/ sentry: https://sentry.ceph.com/ceph/sepia/?query=fa354f2d1ab542d1b97d9ef6d802e2e7 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573006] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573006/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573006/ sentry: https://sentry.ceph.com/ceph/sepia/?query=69330a8e88244ecd8b005af3cd75a0b6 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_msgr' [5573010] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:05:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573010/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573010/ sentry: https://sentry.ceph.com/ceph/sepia/?query=a2f003dc4ee040e7b5fb1ecbefe9b929 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573016] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573016/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573016/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c070ef5b54704a7a992ae95c6d4b85b4 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573035] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} ----------------------------------------------------------------- time: 00:22:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573035/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573035/ sentry: https://sentry.ceph.com/ceph/sepia/?query=96615f65698f48588e0d559ebf50b953 Command failed (workunit test osd/bad-inc-map.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh' [5573041] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573041/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573041/ sentry: https://sentry.ceph.com/ceph/sepia/?query=14e6bb7f07b24c49a50db656c03913b3 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573047] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:06:32 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573047/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573047/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9a54877165184c65ac04c89004cebd22 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573053] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:05:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573053/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573053/ sentry: https://sentry.ceph.com/ceph/sepia/?query=801b67e8115542168f5dbf1cb39d6842 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573057] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/sample_fio} ----------------------------------------------------------------- time: 00:04:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573057/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573057/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5d2350df6a754f768657225ea34a724f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573075] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:05:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573075/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573075/ sentry: https://sentry.ceph.com/ceph/sepia/?query=44af6af13924475386630ccc40e00311 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573079] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573079/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573079/ sentry: https://sentry.ceph.com/ceph/sepia/?query=de64a50100e94407986bc1808454fc42 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573087] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573087/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573087/ sentry: https://sentry.ceph.com/ceph/sepia/?query=359a1406cbe04d44b0f08fef18382bac Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573093] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:05:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573093/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573093/ sentry: https://sentry.ceph.com/ceph/sepia/?query=90913d7c7ad54c25b3b6e41b9e782e9b Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573095] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:05:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573095/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573095/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5e603a62a1c049b0b269ad5b2f7646dd Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573098] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/scrub} ----------------------------------------------------------------- time: 00:05:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573098/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573098/ sentry: https://sentry.ceph.com/ceph/sepia/?query=72428caff4d34184915c331f74a240f4 Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery- scrub.sh' [5573108] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/sample_radosbench} ----------------------------------------------------------------- time: 00:04:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573108/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573108/ sentry: https://sentry.ceph.com/ceph/sepia/?query=aaf2b52939654d0681a94c20e2d7c81f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573110] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:04:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573110/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573110/ sentry: https://sentry.ceph.com/ceph/sepia/?query=cc4c6363689d45dabd593a24366f9ed0 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573116] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:49 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573116/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573116/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c540c3e1c03847b7a69e87785df7b9a3 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573128] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:06:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573128/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573128/ sentry: https://sentry.ceph.com/ceph/sepia/?query=44895f71a4064026b1809de7ca408155 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573146] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:05:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573146/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573146/ sentry: https://sentry.ceph.com/ceph/sepia/?query=e1ed17c5fd0e40a9be1060f4ba0c9aec Command crashed: 'sudo ceph --cluster ceph osd crush tunables jewel' [5573150] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:05:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573150/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573150/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d30d8c07fa254070b7f16fa4632c868e Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573158] rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573158/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573158/ sentry: https://sentry.ceph.com/ceph/sepia/?query=4f6ca2d0f9984da99ee0a508c4aa640c Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573160] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} ----------------------------------------------------------------- time: 00:04:32 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573160/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573160/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5cacba6088cf4cdc8e510fbe96724a3d Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573168] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/crush} ----------------------------------------------------------------- time: 00:05:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573168/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573168/ sentry: https://sentry.ceph.com/ceph/sepia/?query=e406f3fd38354cde8cae5cf7f180c0cb Command failed (workunit test crush/crush-choose-args.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush/crush-choose- args.sh' [5573170] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:13:11 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573170/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573170/ sentry: https://sentry.ceph.com/ceph/sepia/?query=657f4136536d4dc4bef1f3fa096d481c Command crashed: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json' [5573174] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573174/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573174/ sentry: https://sentry.ceph.com/ceph/sepia/?query=71b7f3143bfb4b0c90d81ef6f6b9037f Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573178] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} ----------------------------------------------------------------- time: 00:18:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573178/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573178/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d3a9984620c94e1d99d875caf5780cae Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [5573182] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:04:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573182/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573182/ sentry: https://sentry.ceph.com/ceph/sepia/?query=a2ae2f1be2284ac7aaf0e49d72755417 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573184] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:05:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573184/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573184/ sentry: https://sentry.ceph.com/ceph/sepia/?query=f34e949970494bfeb2e948dc54da5d22 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573186] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:05:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573186/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573186/ sentry: https://sentry.ceph.com/ceph/sepia/?query=cddfc6398c834b49ad14340397d05887 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573187] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:05:16 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573187/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573187/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d1b88947c13444cd9b05b3d785fbbf7c Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573191] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:08:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573191/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573191/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d5d371ead28e4a5580685970d228c5d8 Command failed on smithi142 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' [5573193] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:05:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573193/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573193/ sentry: https://sentry.ceph.com/ceph/sepia/?query=af1a9709471444f99931d7daf6143d8f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573198] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:05:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573198/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573198/ sentry: https://sentry.ceph.com/ceph/sepia/?query=70104206c14a4074afa987f619d49239 Command crashed: 'sudo ceph --cluster ceph osd crush tunables jewel' [5573205] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/cosbench_64K_write} ----------------------------------------------------------------- time: 00:04:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573205/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573205/ sentry: https://sentry.ceph.com/ceph/sepia/?query=4ebc22fa252142ff8e66865e253ca71e Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573206] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:27:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573206/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573206/ "2020-10-30T06:06:31.063518+0000 osd.7 (osd.7) 10 : cluster [ERR] osd.7 found snap mapper error on pg 3.2e oid 3:7664ebeb:::smithi01437199-697:10f snaps missing in mapper, should be: 10f was r -2...repaired" in cluster log [5573211] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:05:35 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573211/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573211/ sentry: https://sentry.ceph.com/ceph/sepia/?query=a933a8ef78724800b882eaaad3ff78be Command crashed: 'sudo ceph --cluster ceph osd crush tunables jewel' [5573213] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573213/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573213/ sentry: https://sentry.ceph.com/ceph/sepia/?query=26ef7f44be0546e494c93faea48e307e Command failed on smithi140 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' [5573218] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:06:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573218/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573218/ sentry: https://sentry.ceph.com/ceph/sepia/?query=10247f5119b641df9e0903019f76c34c Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573219] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:06:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573219/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573219/ sentry: https://sentry.ceph.com/ceph/sepia/?query=3351383a0f1e4417920fa7422897de71 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573221] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:05:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573221/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573221/ sentry: https://sentry.ceph.com/ceph/sepia/?query=b5845d94d43347d7a4b9b6fff8b0f5d3 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573224] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:55 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573224/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573224/ sentry: https://sentry.ceph.com/ceph/sepia/?query=751310e188d848cbb4ee57213c1e33c1 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573225] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573225/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573225/ sentry: https://sentry.ceph.com/ceph/sepia/?query=e2c90d53eada4d3aac21960b6a44d268 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573227] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:18:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573227/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573227/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9d0cd991632e422483c277a3be9268cd Command failed (workunit test mon/caps.sh) on smithi174 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' [5573232] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_read} ----------------------------------------------------------------- time: 00:04:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573232/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573232/ sentry: https://sentry.ceph.com/ceph/sepia/?query=99b5e08305184d31a5c65cc73b844d6f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573236] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:05:16 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573236/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573236/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5c9c1cae43884b75b4e5e4a84886789a Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573237] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:16:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573237/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573237/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c27a0a2a3ef54e58bbb27d676006d43f Command failed (workunit test rados/test.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5573238] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573238/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573238/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5573239] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:05:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573239/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573239/ sentry: https://sentry.ceph.com/ceph/sepia/?query=ec4c6161c2fd4925971df11101dd44ac Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573242] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573242/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573242/ sentry: https://sentry.ceph.com/ceph/sepia/?query=cd3097e4b31249b597e4a6acfbc9a88f Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573243] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:06:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573243/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573243/ sentry: https://sentry.ceph.com/ceph/sepia/?query=61ed4e0c556a4afd9b8c7dba847d6819 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573257] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} ----------------------------------------------------------------- time: 00:04:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573257/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573257/ sentry: https://sentry.ceph.com/ceph/sepia/?query=acfcd03d62544d8f8a1f93a6997a1e5f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573264] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:23:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573264/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573264/ sentry: https://sentry.ceph.com/ceph/sepia/?query=ca714b8895f44ee6853d2c41e1a0ee0a Command failed (workunit test rados/test.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5573266] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:17:05 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573266/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573266/ sentry: https://sentry.ceph.com/ceph/sepia/?query=886b0508e1274bb08b37edcd6b6a1403 Command failed (workunit test rados/test_python.sh) on smithi089 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' [5573270] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:05:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573270/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573270/ sentry: https://sentry.ceph.com/ceph/sepia/?query=0540c54241004623933ee6af9c539357 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573271] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:05:18 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573271/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573271/ sentry: https://sentry.ceph.com/ceph/sepia/?query=6ae9ec345a1d4e2d9df2ef08083e6d65 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573273] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:05:11 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573273/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573273/ sentry: https://sentry.ceph.com/ceph/sepia/?query=48bfd6e718bc4b2baa27638ca170d337 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573283] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_read} ----------------------------------------------------------------- time: 00:04:32 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573283/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573283/ sentry: https://sentry.ceph.com/ceph/sepia/?query=2bcc37b5c4e140cb9bc68b603ab85739 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573284] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:05:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573284/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573284/ sentry: https://sentry.ceph.com/ceph/sepia/?query=577ffb1d3bd54ddc84a28973c854d9a0 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573285] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:06:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573285/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573285/ sentry: https://sentry.ceph.com/ceph/sepia/?query=b36847795da8412db4fce64ed9b346f4 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573291] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:10:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573291/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573291/ sentry: https://sentry.ceph.com/ceph/sepia/?query=0699e40829854b079e1bd78044c55ad5 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573301] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573301/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573301/ sentry: https://sentry.ceph.com/ceph/sepia/?query=ccf75eebe9104a12893e2cf7d3ece412 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573308] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} ----------------------------------------------------------------- time: 00:04:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573308/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573308/ sentry: https://sentry.ceph.com/ceph/sepia/?query=478c300a018c4ca387a2a8c3b1c5636f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573310] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573310/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573310/ sentry: https://sentry.ceph.com/ceph/sepia/?query=57c47decb522446998aa993f769b1e4f Command crashed: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5573314] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:05:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573314/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573314/ sentry: https://sentry.ceph.com/ceph/sepia/?query=6d6b8494c1fa4001acf981dca97068d8 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573316] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/prometheus} ----------------------------------------------------------------- time: 00:05:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573316/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573316/ sentry: https://sentry.ceph.com/ceph/sepia/?query=00fc16ba8fa7453db443f101aae5c878 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573323] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:05:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573323/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573323/ sentry: https://sentry.ceph.com/ceph/sepia/?query=b0b2290b610d438e870605d2fe2e3618 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573335] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/osd} ----------------------------------------------------------------- time: 00:22:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573335/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573335/ sentry: https://sentry.ceph.com/ceph/sepia/?query=26919d1ff5f842d29376a743b29f83f5 Command failed (workunit test osd/bad-inc-map.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh' [5573337] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_write} ----------------------------------------------------------------- time: 00:04:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573337/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573337/ sentry: https://sentry.ceph.com/ceph/sepia/?query=8c539fde6e5e4852ac8e6d7d44870476 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573339] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:06:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573339/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573339/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5189053e7f77483e9830367fe06b7b46 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573346] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573346/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573346/ sentry: https://sentry.ceph.com/ceph/sepia/?query=8328b96836c4493f829e23ba32c8e288 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573347] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:05:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573347/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573347/ sentry: https://sentry.ceph.com/ceph/sepia/?query=6dc23617c73e4d47950fbbd75199fd9e Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573348] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:05:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573348/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573348/ sentry: https://sentry.ceph.com/ceph/sepia/?query=490e94dd5e2641c39545cd2c43144d93 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573350] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573350/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573350/ sentry: https://sentry.ceph.com/ceph/sepia/?query=3156facf487f426ba15a99a92a2010fd Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573358] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:27 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573358/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573358/ sentry: https://sentry.ceph.com/ceph/sepia/?query=b4d36978dcd94c13a3e8d110c737a839 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573362] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} ----------------------------------------------------------------- time: 00:04:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573362/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573362/ sentry: https://sentry.ceph.com/ceph/sepia/?query=56568dbf8f074eeab2fd97305cbbeb4c Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573364] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573364/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573364/ sentry: https://sentry.ceph.com/ceph/sepia/?query=3569a6577b9e4058a8c71c6fd334a11a Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573365] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:05:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573365/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573365/ sentry: https://sentry.ceph.com/ceph/sepia/?query=3a75fa8d2ae048b6a7c217bb49a14165 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573372] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573372/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573372/ sentry: https://sentry.ceph.com/ceph/sepia/?query=72352b8855404955a4f0f2f6bbb7db2b Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573373] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:04:59 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573373/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573373/ sentry: https://sentry.ceph.com/ceph/sepia/?query=ae82b11b991e4507afea19730b4c2c7e Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573375] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:26 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573375/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573375/ sentry: https://sentry.ceph.com/ceph/sepia/?query=287d914361454b36a611da5c50c51c3b Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573385] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:05:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573385/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573385/ sentry: https://sentry.ceph.com/ceph/sepia/?query=bbc7d657c568488087c264cd4e91e0bc Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573386] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573386/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573386/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c7f8b9c85f8242b29b4586d793c11d6b Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573387] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:04:49 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573387/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573387/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c25b1b1e99834af2ba7ccb7221d9ef64 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573389] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:05:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573389/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573389/ sentry: https://sentry.ceph.com/ceph/sepia/?query=15558135dc414f8fbe739a63d83fee11 Command crashed: 'sudo ceph --cluster ceph osd crush tunables jewel' [5573394] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:05:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573394/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573394/ sentry: https://sentry.ceph.com/ceph/sepia/?query=32fa220954c045d983ef31190b99299c Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573396] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573396/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573396/ sentry: https://sentry.ceph.com/ceph/sepia/?query=0567affbd0bb4dfa81e3d75957426697 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573397] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush} ----------------------------------------------------------------- time: 00:05:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573397/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573397/ sentry: https://sentry.ceph.com/ceph/sepia/?query=eb80bd8caee2420b93b7ee2656d06ad7 Command failed (workunit test crush/crush-choose-args.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush/crush-choose- args.sh' [5573398] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:12:43 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573398/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573398/ sentry: https://sentry.ceph.com/ceph/sepia/?query=a296b8c74a104331a7e36abf2dbe0047 Command crashed: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json' [5573402] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:05:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573402/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573402/ sentry: https://sentry.ceph.com/ceph/sepia/?query=f8d13ab2092b44fe9c2cf4c7959225f8 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573403] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:55 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573403/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573403/ sentry: https://sentry.ceph.com/ceph/sepia/?query=1b4420f21c25449e9b0565845ec1a6c7 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573404] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/crash} ----------------------------------------------------------------- time: 00:05:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573404/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573404/ sentry: https://sentry.ceph.com/ceph/sepia/?query=eabfb7f1f1a143ed91eeb3a7adb5f0db Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573407] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:05:05 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573407/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573407/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d6e61aba8c944be586e757e32ae66889 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573408] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:05:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573408/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573408/ sentry: https://sentry.ceph.com/ceph/sepia/?query=035e585ce81647f5bcfe8bc77331e5db Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573409] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:07:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573409/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573409/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9341e0b8c4a644859ff04c6cdd5f4b97 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573412] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:09:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573412/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573412/ sentry: https://sentry.ceph.com/ceph/sepia/?query=39ff5ccabacd45c0bcea932fe07e763a Command failed on smithi078 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' [5573415] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:06:05 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573415/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573415/ sentry: https://sentry.ceph.com/ceph/sepia/?query=7c89ab76865945fbaf4c440413a4810b Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573417] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} ----------------------------------------------------------------- time: 00:05:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573417/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573417/ sentry: https://sentry.ceph.com/ceph/sepia/?query=4c35ad3b3a374af584d170c654c15350 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573419] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:05:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573419/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573419/ sentry: https://sentry.ceph.com/ceph/sepia/?query=a94c424305e0491a81b634dd3e1e6753 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573423] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:05:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573423/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573423/ sentry: https://sentry.ceph.com/ceph/sepia/?query=1c4e622435864caea25958b31e445524 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573424] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573424/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573424/ sentry: https://sentry.ceph.com/ceph/sepia/?query=909c004b211642ce8752c60196286334 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573430] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:16:43 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573430/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573430/ sentry: https://sentry.ceph.com/ceph/sepia/?query=2016aa80bb0a4c10908baa008b3698b5 Command failed (workunit test rados/test.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5573431] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/erasure-code} ----------------------------------------------------------------- time: 00:05:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573431/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573431/ sentry: https://sentry.ceph.com/ceph/sepia/?query=bc4f5fed3501434cb3cef0b1f9089701 Command failed (workunit test erasure-code/test-erasure-code-plugins.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test- erasure-code-plugins.sh' [5573434] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} ----------------------------------------------------------------- time: 00:05:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573434/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573434/ sentry: https://sentry.ceph.com/ceph/sepia/?query=210f6c5d57cb49ff9731dd817a7319de Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573440] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} ----------------------------------------------------------------- time: 00:04:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573440/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573440/ sentry: https://sentry.ceph.com/ceph/sepia/?query=83ad863e46854e67ae319c3f9a359e0f Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573447] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:05:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573447/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573447/ sentry: https://sentry.ceph.com/ceph/sepia/?query=95397584eaa64ebab3508569110d071d Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573450] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:05:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573450/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573450/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5f97ba1f8fae41518e8b426f1fefe888 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573452] rados/singleton/{all/mon-config mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573452/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573452/ sentry: https://sentry.ceph.com/ceph/sepia/?query=bc68f37d3d5546d79b809e15a654c145 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573454] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:05:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573454/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573454/ sentry: https://sentry.ceph.com/ceph/sepia/?query=39c7c9db1c39467896f23469a560de90 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573455] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:05:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573455/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573455/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9285a4cb1d6b49098e0c025bbd3ecd5d Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573456] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} ----------------------------------------------------------------- time: 00:16:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573456/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573456/ sentry: https://sentry.ceph.com/ceph/sepia/?query=24dbb6485f924318bde8176f17c47532 Command failed (workunit test rados/test_python.sh) on smithi150 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' [5573457] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:25:32 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573457/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573457/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5eefbacc882a4868880c533c1a4c7834 Command failed (workunit test rados/test.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=874c7de67fc4e3c0fd6168d3999a388dd662b910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5573459] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:05:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573459/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573459/ sentry: https://sentry.ceph.com/ceph/sepia/?query=f690a327f818412fa58e35968b443a9d Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573463] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573463/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573463/ sentry: https://sentry.ceph.com/ceph/sepia/?query=3850410146a24b03a1749449953d25d6 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573464] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:05:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573464/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573464/ sentry: https://sentry.ceph.com/ceph/sepia/?query=4140c7c1354b458799dd8f9f1a0126a3 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573467] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:05:58 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573467/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573467/ sentry: https://sentry.ceph.com/ceph/sepia/?query=400f2a0200ec47429b14a2ee584a917a Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573469] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_4M_write} ----------------------------------------------------------------- time: 00:04:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573469/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573469/ sentry: https://sentry.ceph.com/ceph/sepia/?query=0bcdf0f5d1f94a0b9dd4d3e6edb57c8c Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573471] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} ----------------------------------------------------------------- time: 00:05:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573471/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573471/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d88789b054a1466ea5db03a7ac49af2c Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573485] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:05:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573485/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573485/ sentry: https://sentry.ceph.com/ceph/sepia/?query=808734d3934a4ca8b8ce8b5a58356013 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573487] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:09:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573487/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573487/ sentry: https://sentry.ceph.com/ceph/sepia/?query=7d1cf4a3e089418b8d89dd39675b5b62 Command crashed: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5573491] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:55 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573491/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573491/ sentry: https://sentry.ceph.com/ceph/sepia/?query=eb35dc865b404724ac5ea99c4bd7a879 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573496] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/radosbench_omap_write} ----------------------------------------------------------------- time: 00:05:01 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573496/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573496/ sentry: https://sentry.ceph.com/ceph/sepia/?query=2391cc6993ee48c1aef4e2f240adf908 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573498] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:05:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573498/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573498/ sentry: https://sentry.ceph.com/ceph/sepia/?query=964a37c8091741c99353baf36c1565d1 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573501] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573501/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573501/ sentry: https://sentry.ceph.com/ceph/sepia/?query=d6d62fac2c0b4d1689fcb954fc0c7e67 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573510] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:05:16 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573510/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573510/ sentry: https://sentry.ceph.com/ceph/sepia/?query=5a775e81dcad4cee9d61ba123b1e87ac Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573515] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:05:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573515/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573515/ sentry: https://sentry.ceph.com/ceph/sepia/?query=045fb60be67449f59ab53a52c8b16b7d Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573516] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:49 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573516/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573516/ sentry: https://sentry.ceph.com/ceph/sepia/?query=8e44bda3ebba4fe59b527bc6414c4297 Command crashed: 'sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force' [5573521] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573521/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573521/ sentry: https://sentry.ceph.com/ceph/sepia/?query=6b115562c3064e15acd9bca0817fbb24 Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' [5573523] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/sample_fio} ----------------------------------------------------------------- time: 00:04:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573523/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573523/ sentry: https://sentry.ceph.com/ceph/sepia/?query=cf791c6122b54d7eb77ba8dbd93d76fa Command crashed: 'sudo ceph --cluster ceph osd crush tunables default' Dead ================================================================= [5573215] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573215/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573215/ Pass ================================================================= [5572976] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:32 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572976/ [5572978] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:55:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572978/ [5572980] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mon} time: 00:42:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572980/ [5572982] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:19:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572982/ [5572984] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/radosbench} time: 01:54:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572984/ [5572992] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:09:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572992/ [5572994] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:37:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572994/ [5572998] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5572998/ [5573004] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:06:35 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573004/ [5573008] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_5925} time: 00:11:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573008/ [5573012] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573012/ [5573014] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:13:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573014/ [5573018] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:15:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573018/ [5573019] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:04:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573019/ [5573021] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:22:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573021/ [5573023] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos} time: 00:04:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573023/ [5573025] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/crash} time: 00:12:52 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573025/ [5573027] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573027/ [5573029] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:28:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573029/ [5573031] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573031/ [5573033] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:23:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573033/ [5573037] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:19:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573037/ [5573039] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:31:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573039/ [5573043] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:30:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573043/ [5573045] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:19:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573045/ [5573049] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573049/ [5573051] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} time: 00:46:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573051/ [5573055] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:14:22 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573055/ [5573059] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:20:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573059/ [5573061] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:30:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573061/ [5573063] rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:28:18 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573063/ [5573065] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573065/ [5573067] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} time: 05:14:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573067/ [5573069] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:12:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573069/ [5573071] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573071/ [5573073] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/connectivity} time: 00:17:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573073/ [5573077] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:33:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573077/ [5573081] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:25:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573081/ [5573083] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:25:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573083/ [5573084] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/rados_striper} time: 00:07:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573084/ [5573089] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:59 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573089/ [5573091] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:21:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573091/ [5573097] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:26:59 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573097/ [5573100] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:20:21 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573100/ [5573102] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:21:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573102/ [5573104] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/failover} time: 00:14:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573104/ [5573106] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:16:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573106/ [5573112] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:27:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573112/ [5573114] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:05:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573114/ [5573118] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:33:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573118/ [5573120] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:29:48 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573120/ [5573122] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:20:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573122/ [5573124] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:14:35 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573124/ [5573126] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:23:05 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573126/ [5573130] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache} time: 00:10:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573130/ [5573132] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573132/ [5573134] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:57 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573134/ [5573136] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:54 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573136/ [5573138] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep} time: 00:19:26 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573138/ [5573140] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:11:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573140/ [5573142] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:31:50 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573142/ [5573144] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:35:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573144/ [5573148] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:13:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573148/ [5573152] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:16:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573152/ [5573154] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:32:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573154/ [5573156] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573156/ [5573162] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:08:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573162/ [5573164] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:32:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573164/ [5573166] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:13:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573166/ [5573172] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573172/ [5573176] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench} time: 02:04:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573176/ [5573180] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect} time: 00:14:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573180/ [5573188] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:12:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573188/ [5573189] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:18:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573189/ [5573190] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:49 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573190/ [5573192] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/insights} time: 00:14:18 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573192/ [5573194] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:11:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573194/ [5573195] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:15:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573195/ [5573196] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573196/ [5573197] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} time: 00:16:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573197/ [5573199] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:24:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573199/ [5573200] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:05:58 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573200/ [5573201] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573201/ [5573202] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 01:23:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573202/ [5573203] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573203/ [5573204] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:07:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573204/ [5573207] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573207/ [5573208] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados tasks/mon_recovery validater/valgrind} time: 00:26:27 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573208/ [5573209] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:35:50 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573209/ [5573210] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:39:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573210/ [5573212] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:23:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573212/ [5573214] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:20:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573214/ [5573216] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:01 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573216/ [5573217] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573217/ [5573220] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:21:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573220/ [5573222] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573222/ [5573223] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:06:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573223/ [5573226] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:06:21 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573226/ [5573228] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:10:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573228/ [5573229] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:29:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573229/ [5573230] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:31:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573230/ [5573231] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:11:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573231/ [5573233] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:26:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573233/ [5573234] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:30:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573234/ [5573235] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} time: 00:12:16 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573235/ [5573240] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr} time: 00:13:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573240/ [5573241] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:27:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573241/ [5573244] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:14:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573244/ [5573245] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:34:43 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573245/ [5573246] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:31:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573246/ [5573247] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:18:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573247/ [5573248] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:29:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573248/ [5573249] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573249/ [5573250] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:05 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573250/ [5573251] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/mon_recovery validater/lockdep} time: 00:11:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573251/ [5573252] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:07:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573252/ [5573253] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/connectivity} time: 00:18:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573253/ [5573254] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:24:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573254/ [5573255] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:07:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573255/ [5573256] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:13:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573256/ [5573258] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} time: 00:10:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573258/ [5573259] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573259/ [5573260] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:38:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573260/ [5573261] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:26:48 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573261/ [5573262] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:09:48 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573262/ [5573263] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:19:05 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573263/ [5573265] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573265/ [5573267] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:46:26 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573267/ [5573268] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/misc} time: 00:28:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573268/ [5573269] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/many workloads/snaps-few-objects} time: 00:32:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573269/ [5573272] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:14:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573272/ [5573274] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/progress} time: 00:19:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573274/ [5573275] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573275/ [5573276] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} time: 00:06:27 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573276/ [5573277] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:11:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573277/ [5573278] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:04:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573278/ [5573279] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:12:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573279/ [5573280] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:30 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573280/ [5573281] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:17:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573281/ [5573282] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:13:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573282/ [5573286] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:14:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573286/ [5573287] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:13:11 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573287/ [5573288] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/readwrite} time: 00:11:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573288/ [5573289] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:29:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573289/ [5573290] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:12:49 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573290/ [5573292] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:21:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573292/ [5573293] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:12:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573293/ [5573294] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:31:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573294/ [5573295] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} time: 00:46:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573295/ [5573296] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:20:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573296/ [5573297] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:31:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573297/ [5573298] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:21:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573298/ [5573299] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:05:59 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573299/ [5573300] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573300/ [5573302] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mon} time: 00:43:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573302/ [5573303] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:24:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573303/ [5573304] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} time: 00:17:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573304/ [5573305] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:27 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573305/ [5573306] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:33:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573306/ [5573307] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:30:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573307/ [5573309] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:33:21 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573309/ [5573311] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:12:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573311/ [5573312] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/one workloads/snaps-few-objects} time: 00:36:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573312/ [5573313] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:31:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573313/ [5573315] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:06:58 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573315/ [5573317] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:12:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573317/ [5573318] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:19:01 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573318/ [5573319] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:12:38 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573319/ [5573320] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:58 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573320/ [5573321] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:44:45 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573321/ [5573322] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/repair_test} time: 00:15:43 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573322/ [5573324] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:33:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573324/ [5573325] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:18:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573325/ [5573326] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573326/ [5573327] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:54:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573327/ [5573328] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573328/ [5573329] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:21 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573329/ [5573330] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:20:27 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573330/ [5573331] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:24:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573331/ [5573332] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:20:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573332/ [5573333] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:14:38 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573333/ [5573334] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:20:49 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573334/ [5573336] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573336/ [5573338] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:26:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573338/ [5573340] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:16:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573340/ [5573341] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:05:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573341/ [5573342] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:25:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573342/ [5573343] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:14:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573343/ [5573344] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:26:08 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573344/ [5573345] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:12:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573345/ [5573349] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:31:52 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573349/ [5573351] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:14:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573351/ [5573352] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} time: 00:15:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573352/ [5573353] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:35:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573353/ [5573354] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/pool-create-delete} time: 00:16:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573354/ [5573355] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-radosbench} time: 02:52:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573355/ [5573356] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:07:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573356/ [5573357] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/workunits} time: 00:07:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573357/ [5573359] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:09:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573359/ [5573360] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:29:26 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573360/ [5573361] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:13:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573361/ [5573363] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:34:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573363/ [5573366] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573366/ [5573367] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/scrub} time: 01:30:18 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573367/ [5573368] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:13:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573368/ [5573369] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:57:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573369/ [5573370] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:22 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573370/ [5573371] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench} time: 01:38:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573371/ [5573374] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573374/ [5573376] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:14:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573376/ [5573377] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:06:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573377/ [5573378] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:12:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573378/ [5573379] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573379/ [5573380] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:50 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573380/ [5573381] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} time: 00:27:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573381/ [5573382] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:13:18 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573382/ [5573383] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:23:06 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573383/ [5573384] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573384/ [5573388] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:22:25 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573388/ [5573390] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:27:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573390/ [5573391] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:13:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573391/ [5573392] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:25:01 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573392/ [5573393] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} time: 00:27:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573393/ [5573395] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:33:57 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573395/ [5573399] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects} time: 00:22:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573399/ [5573400] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} time: 00:12:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573400/ [5573401] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_5925} time: 00:10:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573401/ [5573405] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:38:29 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573405/ [5573406] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:15:50 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573406/ [5573410] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}} time: 00:14:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573410/ [5573411] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:27:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573411/ [5573413] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573413/ [5573414] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:08:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573414/ [5573416] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573416/ [5573418] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:11:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573418/ [5573420] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:18:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573420/ [5573421] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573421/ [5573422] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} time: 00:06:22 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573422/ [5573425] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:27 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573425/ [5573426] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:22:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573426/ [5573427] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:38:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573427/ [5573428] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:09:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573428/ [5573429] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-small} time: 00:15:48 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573429/ [5573432] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:27:46 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573432/ [5573433] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:11 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573433/ [5573435] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps} time: 00:22:35 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573435/ [5573436] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573436/ [5573437] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:53 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573437/ [5573438] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:12:55 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573438/ [5573439] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:19:32 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573439/ [5573441] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_api_tests} time: 00:24:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573441/ [5573442] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} time: 00:20:23 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573442/ [5573443] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:04:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573443/ [5573444] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/failover} time: 00:10:51 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573444/ [5573445] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:24:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573445/ [5573446] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573446/ [5573448] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573448/ [5573449] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache} time: 00:09:12 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573449/ [5573451] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:04:56 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573451/ [5573453] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:10:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573453/ [5573458] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573458/ [5573460] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:30:41 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573460/ [5573461] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mgr} time: 00:10:01 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573461/ [5573462] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:15:43 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573462/ [5573465] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:13:54 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573465/ [5573466] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:19:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573466/ [5573468] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} time: 02:02:48 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573468/ [5573470] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/connectivity} time: 00:16:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573470/ [5573472] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573472/ [5573473] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:12:35 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573473/ [5573474] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:21:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573474/ [5573475] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:14:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573475/ [5573476] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:13:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573476/ [5573477] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:23:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573477/ [5573478] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:31 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573478/ [5573479] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:12:14 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573479/ [5573480] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:15 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573480/ [5573481] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:15:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573481/ [5573482] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:30:44 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573482/ [5573483] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:20:55 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573483/ [5573484] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:55:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573484/ [5573486] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/insights} time: 00:14:07 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573486/ [5573488] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:13:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573488/ [5573489] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:35 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573489/ [5573490] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:21:17 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573490/ [5573492] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:21:33 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573492/ [5573493] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:07:16 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573493/ [5573494] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/misc} time: 00:27:02 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573494/ [5573495] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:22:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573495/ [5573497] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:14:24 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573497/ [5573499] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:23:38 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573499/ [5573500] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:21:13 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573500/ [5573502] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:30:28 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573502/ [5573503] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573503/ [5573504] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:10:34 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573504/ [5573505] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:13:03 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573505/ [5573506] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:29:48 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573506/ [5573507] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch} time: 00:18:47 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573507/ [5573508] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:33:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573508/ [5573509] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:13:19 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573509/ [5573511] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:39 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573511/ [5573512] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:10:42 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573512/ [5573513] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:12:50 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573513/ [5573514] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:31:00 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573514/ [5573517] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:27:09 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573517/ [5573518] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:31:20 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573518/ [5573519] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:38:40 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573519/ [5573520] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:00:04 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573520/ [5573522] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:09:36 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573522/ [5573524] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:37 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573524/ [5573525] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mon} time: 00:44:10 info: http://pulpito.ceph.com/kchai-2020-10-30_02:27:32-rados-wip-kefu-testing-2020-10-29-2343-distro-basic-smithi/5573525/